Part-of-Speech Tagging & Named Entity Recognition

Two classic sequence labeling tasks

Before you can do anything sophisticated with text, you often need to answer two simpler questions: what grammatical role does each word play, and which spans of text refer to real-world things like people, organizations, or dates? These are part-of-speech (POS) tagging and named entity recognition (NER) — two of the oldest and most widely deployed NLP tasks, and both are instances of a more general problem called sequence labeling: given a sequence of tokens, assign one label to each token.

POS tagging assigns each word a grammatical category — noun, verb, adjective, determiner, and so on. "Bank" is a noun in "I went to the bank" and could be a verb in "The plane banked left." NER identifies and classifies spans of text into categories like PERSON, ORGANIZATION, LOCATION, and DATE. In "Marie Curie worked at the Sorbonne in 1898," NER should tag "Marie Curie" as PERSON, "Sorbonne" as ORGANIZATION, and "1898" as DATE.

Why context makes this hard

Both tasks are deceptively simple to describe and genuinely hard to do well, because the correct label for a word almost always depends on its neighbors. "Washington" could be a PERSON (George Washington), a LOCATION (Washington state), or an ORGANIZATION (the Washington administration) depending entirely on surrounding words. A tagger that looks at words in isolation will fail constantly; it needs to model the sequence as a whole.

This is why POS tagging and NER are framed as structured prediction problems rather than independent classification: the labels for adjacent tokens are correlated (a determiner is almost always followed by a noun phrase, and a PERSON entity rarely interrupts mid-token), and the best models explicitly capture that structure instead of tagging each word in a vacuum.

The classical approach: HMMs

The earliest statistical approach to sequence labeling was the Hidden Markov Model (HMM). An HMM assumes there's a hidden sequence of tags generating the observed words, with two sets of probabilities learned from a tagged training corpus: transition probabilities (how likely is tag B to follow tag A?) and emission probabilities (how likely is a given word to appear under a given tag?).

Given a sentence, you then use the Viterbi algorithm — a dynamic programming method — to find the single most probable tag sequence efficiently, rather than checking every possible combination. HMMs work reasonably well but have a real limitation: emission probabilities only look at the current word, so they can't easily use rich features like capitalization, prefixes, suffixes, or surrounding words.

Conditional Random Fields: a better classical model

Conditional Random Fields (CRFs) improved on HMMs by directly modeling the probability of the whole tag sequence given the whole sentence, conditioned on arbitrary features of the input — not just the current word, but capitalization patterns, word shape, whether a word is in a gazetteer of known names, part-of-speech tags from a separate tagger, and more. Because CRFs model the conditional distribution directly rather than generating words from tags, they can incorporate overlapping, hand-engineered features that HMMs can't easily use.

CRFs dominated NER and POS tagging benchmarks for roughly a decade (2003–2013) and are still used in production systems where interpretability, speed, and small training-data requirements matter more than squeezing out the last point of accuracy.

The modern approach: transformers

Transformer-based models reframe sequence labeling as a straightforward application of contextual embeddings. A model like BERT is fine-tuned by adding a simple linear classification layer on top of each token's final hidden state, then trained to predict the BIO tag for that token. Because BERT's self-attention already lets every token see every other token in the sentence, the model learns the contextual disambiguation that HMMs and CRFs had to hand-engineer features for.

This approach — pretrained transformer encoder plus a token-level classification head — now dominates both POS tagging and NER benchmarks, often combined with a CRF layer on top of the transformer output to enforce valid BIO transitions (you can't have an I-LOC immediately after an O, for instance). The CRF didn't disappear; it got demoted from the whole model to a final consistency-enforcing layer.

HMM vs. CRF vs. transformer-based tagging

Why these tasks still matter

It's tempting to think large language models have made POS tagging and NER obsolete — just ask an LLM to extract entities in a prompt. For many applications that's true and convenient. But dedicated taggers remain important where you need: deterministic, auditable behavior (a CRF gives the same output every time, an LLM may not); very low latency at massive scale (search engines tag billions of queries); and tight integration with downstream structured pipelines like dependency parsing, which typically consumes POS tags as an input feature.

What's next

POS tags are also a building block for a deeper structural task: figuring out how words in a sentence relate to each other grammatically. The next lesson, Dependency Parsing, covers how NLP systems represent sentence structure as a tree of head-dependent relations.