Linear Algebra for Machine Learning

Why linear algebra is the language of ML

Every piece of data a model touches — an image, a sentence, a user's click history — gets converted into a vector: a list of numbers. Every operation a model performs — a layer's forward pass, an attention score, a gradient update — is a linear algebra operation on those vectors and the matrices that transform them.

You don't need a full semester of proofs to use this well. You need four things cold: vectors, dot products, matrix multiplication, and eigenvectors/eigenvalues. Everything else in ML math builds on top of these.

Vectors: data as arrows in space

A vector is just an ordered list of numbers, written $\vec{v} = [v_1, v_2, \dots, v_n]$. A word embedding might be a 300-dimensional vector; a grayscale image might be flattened into a 784-dimensional vector (28×28 pixels). The dimension $n$ is simply how many numbers describe one data point.

Two operations matter immediately: addition ($\vec{a} + \vec{b}$, done element-wise) and scalar multiplication ($c\vec{v}$, scaling every entry by $c$). Geometrically, vectors are arrows from the origin, and these operations stretch or combine those arrows.

The dot product: how similarity gets measured

The dot product of two vectors is $\vec{a} \cdot \vec{b} = \sum_i a_i b_i$ — multiply corresponding entries and sum. For $\vec{a} = [2,1,0]$ and $\vec{b} = [1,3,4]$: $(2)(1) + (1)(3) + (0)(4) = 2 + 3 + 0 = 5$.

This single number is doing enormous work across ML. Cosine similarity (used to compare embeddings) is a normalized dot product. A neuron's weighted sum $z = \vec{w} \cdot \vec{x} + b$ is a dot product. Attention scores in a transformer are dot products between query and key vectors. If you understand the dot product, you understand what a huge fraction of ML computation is actually doing: measuring alignment between two vectors.

Matrices: batches of linear transformations

A matrix is a grid of numbers, and multiplying a vector by a matrix applies a linear transformation — a combination of rotation, scaling, and shearing. In a neural network layer, $\vec{y} = W\vec{x} + \vec{b}$ takes an input vector $\vec{x}$, applies the weight matrix $W$, and adds a bias — exactly the dot-product-per-row operation generalized across many output neurons at once.

Matrix multiplication $C = AB$ is defined by $C_{ij} = \sum_k A_{ik}B_{kj}$ — the dot product of row $i$ of $A$ with column $j$ of $B$. This is why matrix shapes must be compatible: the number of columns in $A$ must match the number of rows in $B$.

Eigenvectors and eigenvalues: directions a matrix doesn't rotate

For a square matrix $A$, an eigenvector $\vec{v}$ is a special direction that $A$ only stretches or shrinks — it never rotates it off its own line. Formally: $A\vec{v} = \lambda\vec{v}$, where $\lambda$ (the eigenvalue) is the scaling factor.

For $A = \begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix}$, the vector $[1,0]$ is an eigenvector with eigenvalue 2 (since $A[1,0]^T = [2,0]^T = 2[1,0]^T$), and $[0,1]$ is an eigenvector with eigenvalue 3. Diagonal matrices make this obvious; for general matrices, eigenvectors/eigenvalues still exist but take more work to find (solving $\det(A - \lambda I) = 0$).

Core linear algebra objects and their ML role

What's next

With vectors and matrices as the language and eigenvectors as the way to find a matrix's "natural directions," you have the full linear-algebra toolkit that ML libraries run under the hood. The next lesson — Calculus for Machine Learning — covers the other half of the math stack: how models actually learn by following gradients computed with derivatives.