Feature Engineering for Text Data
Text isn't a feature until you make it one
Raw text — a review, a support ticket, a tweet — can't be fed directly into most ML models, which expect fixed-length numeric vectors. Feature engineering for text is the process of converting variable-length strings into structured representations that preserve the signal a model needs while discarding noise. The right choice depends heavily on the downstream task: a spam classifier cares about different signals than a named entity recognizer or a semantic search system.
Preprocessing: the unglamorous foundation
Before any feature extraction, text typically goes through tokenization (splitting into words or subword units), normalization (lowercasing, Unicode normalization, removing accents where appropriate), stopword removal (dropping high-frequency, low-information words like 'the', 'is' — though this is often skipped for modern transformer-based pipelines, which handle them natively), and stemming/lemmatization (reducing words to a root form: 'running' → 'run'). Stemming is a crude heuristic (Porter stemmer), while lemmatization uses morphological/dictionary knowledge to return a real word ('better' → 'good'). The choice matters: aggressive stemming can conflate words with different meanings, hurting tasks like sentiment analysis where 'running' vs. 'ran' tense might matter.
| Feature type | What it captures | Typical use case |
|---|---|---|
| Bag-of-words / TF-IDF | Word frequency, weighted by informativeness | Classic text classification, baselines |
| N-grams (char/word) | Local word order, misspellings (char n-grams) | Spam detection, language ID, typo-robust search |
| Hand-crafted linguistic | POS tags, sentence length, punctuation density | Readability, authorship, style detection |
| Static word embeddings | Distributional semantic similarity | Feature input for shallow models, similarity |
| Contextual embeddings (BERT etc.) | Context-dependent meaning, full sentence semantics | Modern classification, NER, QA, retrieval |
import re
from sklearn.feature_extraction.text import TfidfVectorizer
def hand_crafted_features(text: str) -> dict:
words = text.split()
return {
"char_count": len(text),
"word_count": len(words),
"avg_word_len": sum(len(w) for w in words) / max(len(words), 1),
"exclamation_count": text.count("!"),
"uppercase_ratio": sum(c.isupper() for c in text) / max(len(text), 1),
"digit_ratio": sum(c.isdigit() for c in text) / max(len(text), 1),
"url_present": int(bool(re.search(r"https?://", text))),
}
# TF-IDF with word unigrams + bigrams, and character n-grams for typo robustness
word_vectorizer = TfidfVectorizer(ngram_range=(1, 2), min_df=3, max_features=20000)
char_vectorizer = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), max_features=10000)
X_word = word_vectorizer.fit_transform(corpus)
X_char = char_vectorizer.fit_transform(corpus)
# Hand-crafted + TF-IDF features are often concatenated and fed to a linear
# model (logistic regression, SVM) as a strong, interpretable baseline.From sparse counts to dense embeddings
Bag-of-words and TF-IDF produce sparse, high-dimensional vectors (one dimension per vocabulary term) that ignore word order and semantic similarity ('great' and 'excellent' are orthogonal dimensions despite meaning similar things). Static word embeddings (word2vec, GloVe) address the semantic gap by mapping each word to a dense vector learned from co-occurrence statistics, so semantically similar words land close together in vector space — but a word still gets exactly one vector regardless of context ('bank' the river vs. 'bank' the institution are identical). Contextual embeddings from transformer encoders (BERT, RoBERTa) solve this by producing a different vector for the same word depending on its surrounding sentence, at the cost of requiring a full forward pass through a large model rather than a dictionary lookup.
TF-IDF baselines are still worth building first
Feature leakage hides easily in text
What's Next?
TF-IDF is simple enough to deserve its own deeper look at the weighting scheme itself — why inverse document frequency works, and where it breaks down.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.