k-Nearest Neighbors is almost embarrassingly simple to state: to classify a new point, find the k labeled points closest to it, and let them vote. The majority class among those k neighbors becomes the prediction. For regression, average their target values instead of voting.

There's no training phase in the usual sense — no weights are learned, no loss function is minimized. kNN just memorizes the entire training set and does all its work at prediction time. This makes it the canonical example of lazy learning, in contrast to eager learning methods like linear regression or neural networks, which do the hard work upfront during training and leave prediction cheap.

Distance metrics: how 'nearest' gets defined

Everything about kNN hinges on what 'nearest' means, and that is a choice you make, not a fact about the data.

Euclidean distance — straight-line distance, sqrt(sum((xi - yi)^2)) — is the default and works well when features are on comparable, continuous scales.

Manhattan distance — sum(|xi - yi|) — sums absolute differences along each axis. It's less sensitive to outliers in any single dimension and often preferred in high-dimensional or grid-like spaces.

Cosine distance measures the angle between vectors rather than their magnitude, which makes it the standard choice for text embeddings and other high-dimensional representations where direction matters more than scale.

Hamming distance counts differing positions between two vectors of categorical or binary values — useful for categorical features or bit strings.

The choice of metric is a modeling decision with real consequences; a dataset with features on wildly different scales (income in dollars, age in years) will have Euclidean distance completely dominated by the large-scale feature unless you standardize first.

Note

Because kNN is entirely distance-based, unscaled features with large numeric ranges will dominate the distance calculation and effectively make the model ignore smaller-scale features. Standardize (zero mean, unit variance) or min-max scale every feature before computing distances — this is not optional for kNN the way it is for some other algorithms.

python

Choosing k: the bias-variance tradeoff in one knob

k is the single hyperparameter, and it controls the classic bias-variance tradeoff directly.

Small k (e.g. k=1) makes predictions highly sensitive to individual training points — low bias, high variance. The decision boundary becomes jagged and can overfit to noise; a single mislabeled training point can flip predictions for an entire neighborhood.

Large k smooths the decision boundary by averaging over more neighbors — higher bias, lower variance. Taken to the extreme, if k equals the size of the training set, every prediction is just the overall majority class, ignoring the input entirely.

The standard approach is to try a range of k values and pick the one that maximizes cross-validated accuracy. It's also common practice to use an odd k for binary classification to avoid tie votes.

python

Note

In high-dimensional spaces, the concept of 'nearest' starts to break down. As dimensionality grows, the distance between the closest and farthest points in a dataset shrinks toward zero relative to the typical distance between points — everything becomes roughly equidistant. This means kNN's core assumption, that nearby points are meaningfully more similar than far points, erodes as you add features. Beyond roughly 10-20 dimensions (depending on the data), kNN's accuracy often degrades sharply unless you apply dimensionality reduction (PCA, or domain-specific feature selection) first.

kNN vs. eager learning models

Property	kNN (lazy learning)	Logistic regression / neural nets (eager)
Training time	Near zero — just stores the data	Can be substantial
Prediction time	Slow — compares to all stored points	Fast — just a forward pass
Memory usage	Must store the entire training set	Only stores learned parameters
Decision boundary	Flexible, non-parametric	Shape fixed by model form
Behavior in high dimensions	Degrades — curse of dimensionality	More robust, with regularization

When kNN is a good choice

kNN shines on smaller, lower-dimensional datasets where the decision boundary is genuinely irregular and you don't want to assume a particular functional form. It's commonly used as a fast baseline, for recommendation systems (find users or items similar to a given one), and for anomaly detection (points with no close neighbors are candidate outliers). For large datasets, approximate nearest-neighbor structures like KD-trees or ball trees (used internally by sklearn) make prediction tractable without scanning every stored point.

What's next

kNN makes almost no assumptions about the relationship between features and target, which is powerful but comes at the cost of needing lots of data and suffering in high dimensions. Ridge & Lasso Regression take the opposite approach: assume a linear relationship, but control overfitting through regularization instead of through neighbor averaging.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →