Parameter-Efficient Fine-Tuning (PEFT)
Why full fine-tuning became impractical
Full fine-tuning updates every parameter in a model, which for a modern LLM means storing full gradients and optimizer states (e.g., Adam's two moment buffers) for every single weight — easily 3-4x the model's own memory footprint on top of the model itself. For a 70-billion-parameter model, that can mean needing hundreds of gigabytes of accelerator memory just to fine-tune, and a completely separate full-size model checkpoint for every fine-tuned variant you want to keep around. Parameter-Efficient Fine-Tuning (PEFT) methods update only a small added or selected subset of parameters — often under 1% of the total — while leaving the bulk of the pretrained weights frozen, dramatically cutting memory, storage, and in some cases compute.
LoRA: the dominant PEFT method
Low-Rank Adaptation (LoRA) freezes the original weight matrix $W_0$ and adds a trainable low-rank update: $W = W_0 + BA$, where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, with rank $r$ typically between 4 and 64 — far smaller than $d$ or $k$. Only $A$ and $B$ are trained; $W_0$ never changes. The number of trainable parameters drops from $d \times k$ to $r(d+k)$, often a reduction of 100-1000x for typical transformer weight matrix sizes. At inference, $BA$ can be merged back into $W_0$ with zero added latency, or kept separate to allow swapping between multiple task-specific adapters on top of one shared base model.
import torch
import torch.nn as nn
class LoRALinear(nn.Module):
def __init__(self, base_layer: nn.Linear, rank=8, alpha=16):
super().__init__()
self.base = base_layer
for p in self.base.parameters():
p.requires_grad = False # freeze original weights
d_in, d_out = base_layer.in_features, base_layer.out_features
self.A = nn.Parameter(torch.randn(rank, d_in) * 0.01)
self.B = nn.Parameter(torch.zeros(d_out, rank)) # zero-init -> no change at start
self.scale = alpha / rank
def forward(self, x):
base_out = self.base(x)
lora_out = (x @ self.A.T) @ self.B.T
return base_out + self.scale * lora_out
# Trainable params: only A and B, often <1% of the base layer's parameter countPEFT method families
| Method | What's added/trained | Inference overhead | Typical trainable params |
|---|---|---|---|
| LoRA | Low-rank matrices A, B per weight matrix | None if merged; small if kept separate | 0.1–1% of model |
| Prefix tuning | Trainable virtual tokens prepended to each layer's keys/values | Small — extra tokens in attention | ~0.1–3% of model |
| Adapters (bottleneck layers) | Small inserted feed-forward modules between transformer layers | Small — added forward compute per layer | ~0.5–5% of model |
| BitFit | Only bias terms | None | <0.1% of model |
| Full fine-tuning | Every parameter | None | 100% |
Why low-rank updates work at all
Rank and target-module choice still matter
The practical payoff
Because PEFT keeps the base model frozen and shared, teams can maintain many small task-specific adapters — a few megabytes each, instead of a full multi-gigabyte model copy per task — and swap them in and out on top of a single loaded base model. This makes serving dozens or hundreds of fine-tuned variants economically realistic in a way full fine-tuning never was.
What's next?
LoRA addresses the memory cost of fine-tuning parameters, but the frozen base model itself still needs to be loaded and run — QLoRA pushes efficiency further by combining low-rank adaptation with aggressive quantization of that frozen base, letting large models be fine-tuned on hardware that couldn't otherwise hold them at all.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.