Adapters & Prefix Tuning

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Before LoRA: the earlier PEFT approaches

LoRA dominates current parameter-efficient fine-tuning discussions, but it was preceded by two other influential families worth understanding on their own terms, both because they illustrate different design philosophies for the same problem and because variants of both are still used in specific settings today: adapter modules, which insert small new layers into the network, and prefix/prompt tuning, which adds trainable 'virtual tokens' rather than trainable weights.

Adapter modules

Adapters, introduced for transfer learning in NLP before LoRA existed, insert small bottleneck feed-forward blocks after specific sublayers (typically after attention and after the feed-forward block in each transformer layer). A typical adapter down-projects the hidden dimension $d$ to a much smaller bottleneck dimension $r$, applies a nonlinearity, then up-projects back to $d$, with a residual connection around the whole thing: $h' = h + W_{up}\,\sigma(W_{down}\,h)$. Only the adapter's own parameters ($W_{down}$, $W_{up}$) are trained; everything else in the transformer stays frozen. Unlike LoRA, adapters add a genuine extra computation step in the forward pass — they can't be algebraically merged away at inference time the way LoRA's low-rank update can be folded back into the original weight matrix.

Prefix tuning and prompt tuning

Prefix tuning takes a different approach entirely: instead of modifying any weights, it prepends a sequence of trainable continuous vectors to the key and value sequences at every transformer layer's attention mechanism. These 'virtual tokens' have no corresponding real word — they're pure learned parameters that the attention mechanism attends to just like real context, steering the model's behavior purely through what effectively becomes a learned, soft prompt built into every layer. Prompt tuning is a simpler, shallower cousin that only prepends trainable vectors at the input embedding layer, rather than at every layer — cheaper to train but generally less expressive for larger models.

prefix_tuning_sketch.py

python

import torch, torch.nn as nn

class PrefixAttention(nn.Module):
    def __init__(self, attn_layer, num_prefix_tokens=20, num_heads=12, head_dim=64):
        super().__init__()
        self.attn = attn_layer
        for p in self.attn.parameters():
            p.requires_grad = False   # base attention weights frozen

        # Trainable virtual key/value vectors, prepended at every layer
        self.prefix_k = nn.Parameter(torch.randn(num_prefix_tokens, num_heads, head_dim) * 0.01)
        self.prefix_v = nn.Parameter(torch.randn(num_prefix_tokens, num_heads, head_dim) * 0.01)

    def forward(self, hidden_states, real_k, real_v, **kwargs):
        # Concatenate trainable prefix k/v in front of the real sequence's k/v
        k = torch.cat([self.prefix_k.unsqueeze(0).expand(hidden_states.size(0), -1, -1, -1), real_k], dim=1)
        v = torch.cat([self.prefix_v.unsqueeze(0).expand(hidden_states.size(0), -1, -1, -1), real_v], dim=1)
        return self.attn(hidden_states, k, v, **kwargs)

Adapters vs. prefix/prompt tuning vs. LoRA

Method	What's trained	Mergeable into base weights?	Typical expressiveness vs. size
Adapters	Small inserted feed-forward blocks per layer	No — adds real inference compute	Good; scales with bottleneck size
Prefix tuning	Trainable key/value vectors per layer	No — effectively lengthens every attention computation	Good for larger models; harder to optimize for small ones
Prompt tuning	Trainable input embeddings only	No — adds tokens to input sequence	Weaker than prefix tuning, especially at smaller model scale
LoRA	Low-rank weight deltas	Yes — zero inference overhead once merged	Strong, and now the default for most LLM fine-tuning

Why LoRA mostly won out

LoRA's practical advantage over adapters and prefix tuning isn't necessarily raw expressiveness — in head-to-head comparisons, results are often close. It's deployment ergonomics: a LoRA update can be merged into the base weights with simple matrix addition, producing a model with identical architecture and zero added inference latency, which adapters and prefix tuning structurally cannot do since they add real forward-pass computation that persists at inference time.

Prefix tuning can be harder to optimize

Trainable virtual tokens interact with the attention mechanism in a way that's less directly analogous to the model's existing weight structure than LoRA's additive matrix update, and prefix tuning has historically been reported as more sensitive to initialization and learning rate, particularly on smaller models, than LoRA tends to be.

What's next?

These methods all answer 'how do you adapt one model cheaply' — a related but distinct engineering challenge is adapting or training models that are too large to fit on a single device at all, which is where the various forms of model and pipeline parallelism come in.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →