LoRA Merging & Model Merging

Why you'd want to combine fine-tunes at all

Fine-tuning produces a model specialized for one task: a LoRA adapter trained on legal documents, another trained on customer support tone, another trained on a coding style. Training a separate full model for every combination of skills you want — legal-aware and support-toned and code-fluent all at once — is expensive and doesn't scale as the number of desired combinations grows.

Model merging and LoRA merging address this by combining the weights of multiple already-trained models or adapters into a single set of weights, without any additional training. The goal is a model that exhibits the combined behavior of its ingredients, inferred purely from how their weights relate to each other — no gradient descent required at merge time.

LoRA adapters: why they're mergeable in the first place

LoRA (Low-Rank Adaptation) fine-tuning doesn't modify a model's full weight matrices directly. Instead, for each weight matrix being adapted, it trains a pair of small low-rank matrices whose product approximates the *change* that should be applied to the original weights. The adapter is just that small delta — typically a tiny fraction of the full model's parameter count.

This structure is exactly what makes LoRA adapters easy to combine: because each adapter represents an additive delta on top of a shared frozen base model, you can add multiple deltas together (optionally weighted) and apply the sum to the same base model. You're not reconciling two different full models — you're summing several small, compatible correction terms applied to one.

Model soups: averaging full models, not just adapters

Model soups apply a similar idea at the level of full model weights rather than small adapters: take several independently fine-tuned versions of the *same base model* (e.g., several runs with different hyperparameters or data orderings) and average their weights directly, parameter by parameter.

Counter-intuitively, this often produces a merged model that performs *better* than any single ingredient model on held-out evaluation, provided the models being averaged started from the same pretrained checkpoint and didn't diverge too far from each other during fine-tuning. The averaging acts similarly to ensembling, smoothing out idiosyncratic overfitting from any single training run, but without the inference-time cost of running multiple models — you get one model, not an ensemble you have to run multiple times.

The key constraint is that model soups work well mainly when the models being merged are fine-tuned variants of the same base model with the same architecture. Averaging the weights of unrelated models trained from different initializations generally produces nonsense, because there's no guarantee that corresponding weight positions represent the same learned function across models trained independently.

Common merging techniques beyond simple averaging

Linear weighted averaging — the simplest approach: assign a weight to each model/adapter and take the weighted sum, as shown above. Works reasonably well when the ingredients are similar and not too numerous.

TIES-merging — resolves sign conflicts between models before averaging: for each parameter, it identifies which models agree on the direction of change, discards the ones that disagree (trims low-magnitude, conflicting changes), and merges only the consistent signal. This generally outperforms naive averaging when merging more than two or three models.

DARE (Drop And REscale) — randomly zeroes out a large fraction of the delta parameters in each model before merging, then rescales the remaining ones to compensate. This sparsifies each contribution before combining, which empirically reduces interference between merged models.

SLERP (Spherical Linear Interpolation) — interpolates between two models' weights along a spherical path rather than a straight line, which tends to preserve each model's output distribution characteristics better than linear averaging for a two-model blend.

Merging approaches compared

What's next

Merging changes what a model knows and how it behaves without changing its size. A separate, equally practical concern for deployment is making a model smaller and faster to run — the next lesson compares GGUF, AWQ, and GPTQ, the most common quantization formats for running LLMs efficiently on local and edge hardware.