Named Entity Linking
Beyond recognizing names — identifying who they actually are
Named Entity Recognition (NER) finds spans of text that refer to entities ('Paris', 'Apple', 'Washington') and tags their type (location, organization, person). But NER alone can't tell you whether 'Washington' means the city, the state, or George Washington — and two different documents mentioning 'Apple' might refer to the fruit or the company. Named Entity Linking (NEL), also called entity disambiguation, resolves recognized mentions to specific, unique entries in a knowledge base (Wikidata, Wikipedia, a proprietary company/product catalog), grounding text in a concrete, unambiguous referent.
The pipeline: mention detection, candidate generation, disambiguation
NEL is typically broken into three stages. Mention detection: identify entity spans (often reusing NER, though NEL systems sometimes detect mentions more broadly, including common nouns that could refer to entities). Candidate generation: for each mention string, retrieve a shortlist of plausible knowledge-base entities it could refer to — 'Washington' might retrieve the U.S. state, the city, the first U.S. president, and dozens of people sharing the surname, typically via an alias/surface-form index built from Wikipedia redirects and anchor text. Disambiguation (entity ranking): score each candidate against the mention's context and select the best match, using features like textual similarity between the mention's surrounding sentence and the candidate entity's description, and coherence — whether the set of entities linked across the whole document are mutually consistent (if the document also mentions 'Mount Rainier' and 'Seattle', that favors Washington-the-state over Washington-the-president).
# Simplified disambiguation scoring combining context similarity and
# document-level entity coherence (a common NEL design pattern).
def score_candidate(mention_context_emb, candidate, doc_entities, kb):
# 1. Local context similarity: how well does the candidate's description
# match the surrounding text of the mention?
context_score = cosine_similarity(mention_context_emb, kb.embed(candidate.description))
# 2. Prior popularity: how often is this mention string linked to this
# candidate across the whole training corpus (Wikipedia anchor stats)?
prior_score = kb.link_prior(candidate, mention_string)
# 3. Coherence: does this candidate relate (via KB graph edges) to other
# entities already confidently linked elsewhere in this document?
coherence_score = sum(
kb.related(candidate, other) for other in doc_entities if other.confidence > 0.8
)
return 0.5 * context_score + 0.2 * prior_score + 0.3 * coherence_score
best_candidate = max(candidates, key=lambda c: score_candidate(ctx_emb, c, doc_entities, kb))| Challenge | Why it's hard | Mitigation |
|---|---|---|
| Ambiguous surface forms | Same string, many possible entities ('Washington') | Context similarity + document coherence |
| NIL entities | Mention refers to something not in the KB at all | NIL-prediction threshold / out-of-KB detection |
| Rare/long-tail entities | Few training examples, sparse KB description | Entity embeddings pretrained on KB structure (graph-based) |
| Cross-lingual mentions | Mention language differs from KB's primary language | Multilingual entity embeddings, cross-lingual aliases |
NIL prediction: knowing when there's no answer
A frequently overlooked but critical part of NEL is correctly identifying when a mention refers to an entity not present in the knowledge base at all — called a NIL prediction. A naive system that always picks the best-scoring candidate, even when none is actually a good match, will confidently mislink novel or obscure entities to the nearest plausible KB entry. Well-designed NEL systems include a calibrated 'none of the above' threshold, often trained as a binary decision on top of the top candidate's score, so the system can abstain rather than force a wrong link.
Document-level coherence is what separates NEL from simple string matching
Knowledge base staleness silently degrades linking quality
What's Next?
Linking entities to a knowledge base is only half the picture of structured information extraction — the other half is determining how the entities found in a sentence actually relate to one another.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.