Gaussian Mixture Models

From hard assignments to probabilities

K-Means makes a binary decision for every point: it belongs to exactly one cluster, period. But real data is rarely that clean — a point sitting between two cluster centers is genuinely ambiguous, and forcing a hard choice throws away useful information.

Gaussian Mixture Models (GMMs) replace hard cluster assignment with soft clustering: instead of saying 'this point belongs to cluster 2,' a GMM says 'this point has a 70% probability of belonging to cluster 2 and a 30% probability of belonging to cluster 1.' The model assumes the data was generated by a mixture of several Gaussian (normal) distributions, each with its own mean, covariance, and mixing weight, and the job of fitting a GMM is to recover those distributions from the observed data.

What a Gaussian cluster actually looks like

A single Gaussian distribution in two or more dimensions is defined by a mean vector (its center) and a covariance matrix (which controls its shape — how stretched and in what direction it is). K-Means implicitly assumes every cluster is a perfect sphere, because it uses plain Euclidean distance. A GMM has no such restriction: its covariance matrix lets each cluster be an ellipse of any orientation and elongation, which means GMMs can correctly model clusters that are stretched, tilted, or of very different sizes — shapes that K-Means systematically gets wrong.

Fitting a GMM: the EM algorithm

GMMs are fit using the Expectation-Maximization (EM) algorithm, which alternates between two steps, much like K-Means alternates between assignment and update — but with probabilities instead of hard choices.

E-step (Expectation): For each point, compute the probability it belongs to each of the k Gaussian components, given the current parameter estimates. This produces a soft 'responsibility' value for every point-cluster pair, rather than a hard label.

M-step (Maximization): Using those soft responsibilities as weights, re-estimate each Gaussian's mean, covariance, and mixing weight — essentially a weighted version of fitting a Gaussian distribution, where each point contributes to every cluster's statistics in proportion to its responsibility for that cluster.

Repeat E and M steps until the parameters stop changing meaningfully. Like K-Means, EM is only guaranteed to converge to a local optimum, so running it from multiple random initializations and keeping the best result (by log-likelihood) is standard practice.

Choosing the number of components

Just as K-Means needs k chosen in advance, a GMM needs the number of components chosen in advance. Because GMMs are fully probabilistic models, you can use proper model-selection criteria instead of the elbow method: the Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) both balance model fit (log-likelihood) against model complexity (number of parameters), penalizing models that add components without a corresponding gain in explanatory power. BIC penalizes complexity more heavily than AIC and is generally preferred for choosing the number of mixture components.

Contrast with hard clustering — see how K-Means draws its boundaries

K-Means vs. Gaussian Mixture Models

What's next

GMMs model clusters probabilistically for unsupervised problems. Linear Discriminant Analysis takes a related idea — modeling classes as distributions — but applies it to supervised classification, using class labels to find the projection that best separates known groups.