Language Identification Models

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The first step before any language-specific processing

Before you can apply the right tokenizer, the right stopword list, or route text to the right translation model, you need to know what language it's written in. Language identification (LID) classifies a piece of text into one of dozens to hundreds of possible languages, and it's deceptively tricky for short text, code-switched text (mixing two languages in one sentence), and closely related language pairs that share significant vocabulary (Spanish vs. Portuguese, Indonesian vs. Malay, Croatian vs. Serbian).

Character n-gram statistics: the classical, still-strong baseline

The classic approach, implemented in tools like langdetect and fastText's language identification model, represents text as character n-gram frequency profiles and compares them against precomputed per-language profiles — languages have distinctive, stable character and n-gram distributions (English favors certain letter-pair frequencies; languages with different scripts are trivially distinguished by character set alone). This is extremely fast, requires no neural inference, and works remarkably well for documents of reasonable length (a paragraph or more) in well-resourced languages.

language_id.py

python

import fasttext

# fastText's pretrained LID model: trained on character n-grams across
# 176 languages, fast enough to run on every document in a large corpus.
model = fasttext.load_model("lid.176.bin")

def identify_language(text: str, k: int = 3):
    labels, probs = model.predict(text.replace("\n", " "), k=k)
    return [(label.replace("__label__", ""), float(p)) for label, p in zip(labels, probs)]

print(identify_language("Bonjour, comment allez-vous?"))
# -> [('fr', 0.98), ('ht', 0.01), ('ca', 0.004)]   -- French, with alternatives

print(identify_language("No se si esto es espanol o portugues"))
# -> closely related languages will show up as competitive alternatives,
#    reflecting genuine model uncertainty rather than a bug.

Why short text and code-switching break the simple approach

N-gram statistics need enough text to build a reliable frequency profile — a three-word tweet or a single search query provides far less signal than a full paragraph, causing confidence and accuracy to drop sharply on short inputs. Code-switching — a single message mixing two languages, common in multilingual communities and social media ('I'm so tired pero necesito terminar esto') — breaks the fundamental assumption that a text has one dominant language at all; a global LID label is simply the wrong granularity, and token-level or span-level language tagging is needed instead, treating it more like a sequence labeling problem than single-label classification.

Scenario	Challenge level	Recommended approach
Long, monolingual document	Easy	Character n-gram model (fastText, langdetect)
Short text / single query	Moderate	N-gram model with explicit low-confidence fallback
Closely related language pair	Hard	Specialized discriminative model trained on that pair
Code-switched text	Hard — wrong problem framing entirely	Token/span-level language tagging, not document-level

LID accuracy numbers from benchmarks rarely reflect production text

Published LID benchmarks typically evaluate on clean, well-formed, monolingual text (news articles, Wikipedia). Production text — chat messages, search queries, user-generated content — is shorter, noisier, and far more likely to be code-switched or contain language-ambiguous content (brand names, URLs, numbers). Always validate LID accuracy on a sample of your actual production text distribution before trusting a benchmark-reported accuracy number.

Low-resource languages are systematically underserved

Most pretrained LID models are trained on web-crawled data, which heavily over-represents high-resource languages (English, Spanish, Chinese, French) and under-represents low-resource languages, leading to lower accuracy and smaller language coverage for languages spoken by hundreds of millions of people but less represented in web text. If your application serves users in underrepresented languages, dedicated evaluation (and possibly a specialized or fine-tuned LID model) is necessary rather than assuming off-the-shelf coverage is adequate.

What's Next?

Having covered the breadth of classical and modern NLP tasks for understanding human language, a natural adjacent frontier is how these same modeling principles apply to an entirely different kind of formal language — programming code.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →