Linear Discriminant Analysis (LDA)

A projection built for separation, not variance

Linear Discriminant Analysis answers a specific question: given labeled data from several classes, what single direction (or set of directions) in feature space best separates those classes? It finds this by looking for a linear combination of features that maximizes the separation between class means relative to the spread within each class.

More formally, LDA maximizes the ratio of between-class variance to within-class variance:

J(w) = (w^T * S_between * w) / (w^T * S_within * w)

where w is the projection direction, S_between captures how far apart the class means are, and S_within captures how spread out each class is around its own mean. The direction that maximizes this ratio is one along which classes are as far apart as possible relative to how much they overlap — exactly what you want for both classification and visualization.

LDA as a classifier

LDA assumes each class's data is drawn from a Gaussian distribution, and critically, assumes all classes share the same covariance matrix — they can be centered at different points but have the same shape and spread. Under this assumption, the boundary that optimally separates two classes turns out to be exactly linear, which is where the name comes from.

To classify a new point, LDA computes which class's Gaussian distribution assigns it the highest probability (accounting for each class's prior frequency), and because of the shared-covariance assumption, this reduces to a simple linear discriminant function rather than requiring the point to be checked against a full, separate Gaussian for each class.

LDA vs. PCA: supervised vs. unsupervised axes

LDA and PCA are both linear dimensionality reduction techniques and are frequently confused, but they optimize for completely different things.

PCA is unsupervised. It finds the directions of maximum variance in the data, with no knowledge of class labels at all. The resulting axes capture whatever varies most, which may or may not align with what separates the classes you care about.

LDA is supervised. It explicitly uses class labels to find the directions that best separate known classes, even if those directions don't correspond to the directions of highest overall variance.

A concrete failure mode for PCA: if the direction of maximum variance in the data happens to run parallel to class boundaries rather than across them, PCA will keep that high-variance direction and can discard the lower-variance direction that actually separates the classes — actively hurting a downstream classifier. LDA cannot make this mistake because it is explicitly optimizing for separation, not variance.

LDA vs. PCA

LDA vs. logistic regression

Both LDA and logistic regression produce linear decision boundaries for classification, and in practice they often perform similarly — but they get there differently and have different strengths.

Logistic regression directly models the probability of each class given the input, making no assumption about how the inputs themselves are distributed. LDA instead models how the inputs are distributed within each class (as Gaussians with shared covariance) and derives the classification rule from that generative model.

This difference matters in practice: when the Gaussian assumption roughly holds and classes are well separated, LDA tends to be more stable with smaller sample sizes, since it's using more structure (the full distributional assumption) to estimate its parameters. Logistic regression tends to be more robust when the Gaussian assumption is clearly violated, since it never relies on it. LDA also naturally extends to more than two classes without modification, while logistic regression needs a multinomial extension to do the same.

What's next

LDA uses labeled classes to find meaningful structure. The next lesson, Anomaly Detection in Machine Learning, tackles the opposite problem: finding the rare points that don't fit any known structure at all, often with few or no labels to guide the search.