Text Classification Pipelines
The task that touches almost every NLP application
Text classification — assigning one or more predefined labels to a piece of text — underlies spam filtering, sentiment analysis, topic tagging, intent detection, content moderation, and support-ticket routing. The modeling techniques vary widely, but the pipeline architecture around them is remarkably consistent: ingest text, extract features, classify, and — critically for production systems — monitor and retrain.
Three generations of modeling approach
Classical ML: TF-IDF or bag-of-words features feed into logistic regression, linear SVM, or Naive Bayes. Fast to train, interpretable, strong baseline, but blind to word order and context. Sequence models: word embeddings feed into an LSTM/GRU or a 1D CNN that captures local word order and longer-range dependencies, trained end-to-end. Meaningfully better on tasks where context matters (negation: 'not good' vs. 'good'), at higher compute cost and lower interpretability. Transformer fine-tuning: a pretrained encoder (BERT, RoBERTa, DeBERTa) is fine-tuned with a classification head on top of the [CLS] token or pooled output. This is the current state of the art for most classification tasks with enough labeled data, and increasingly, zero-shot/few-shot prompting of a large language model is used instead of fine-tuning at all when labeled data is scarce.
| Approach | Data needed | Latency/cost | Typical accuracy ceiling |
|---|---|---|---|
| TF-IDF + linear model | Hundreds to low thousands of labels | Very low | Good for simple, lexical-cue tasks |
| LSTM/CNN on embeddings | Thousands of labels | Low-moderate | Better on context-sensitive tasks |
| Fine-tuned transformer | Hundreds to thousands (with pretraining) | Moderate (GPU for training, fast inference) | State of the art for most tasks |
| LLM zero/few-shot prompting | Zero to a handful of examples | High per-request cost, no training | Strong for novel/rare label sets |
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
from datasets import Dataset
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained(
"distilbert-base-uncased", num_labels=3 # e.g., negative/neutral/positive
)
def tokenize(batch):
return tokenizer(batch["text"], truncation=True, padding="max_length", max_length=256)
train_ds = Dataset.from_dict({"text": train_texts, "label": train_labels}).map(tokenize, batched=True)
eval_ds = Dataset.from_dict({"text": eval_texts, "label": eval_labels}).map(tokenize, batched=True)
args = TrainingArguments(
output_dir="./clf",
num_train_epochs=4,
per_device_train_batch_size=16,
learning_rate=2e-5,
weight_decay=0.01,
eval_strategy="epoch",
load_best_model_at_end=True,
)
trainer = Trainer(model=model, args=args, train_dataset=train_ds, eval_dataset=eval_ds)
trainer.train()Multi-label, hierarchical, and extreme classification
Not every classification problem fits the clean 'one text, one label' mold. Multi-label classification allows multiple simultaneous labels (a support ticket can be both 'billing' and 'urgent') and requires a sigmoid output per label with binary cross-entropy loss rather than softmax. Hierarchical classification exploits label taxonomies (a product category tree) by predicting coarse-to-fine, often improving accuracy on rare fine-grained labels by letting the model first narrow down the broad category. Extreme classification deals with label spaces in the thousands to millions (e.g., Wikipedia category tagging), where a standard softmax over all labels becomes computationally prohibitive, motivating techniques like label embedding trees or approximate nearest-neighbor label retrieval.
Class imbalance is the default, not the exception
Fine-tuned models drift as language drifts
What's Next?
One of the most common and commercially important specializations of text classification deserves its own detailed treatment: determining the emotional polarity of a piece of text.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.