Retrieval-Augmented Fine-Tuning (RAFT)

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

The gap RAFT is designed to close

Standard RAG retrieves relevant documents at inference time and feeds them to a model that was never specifically trained to use retrieved context well — it has to generalize from whatever instruction-following ability it picked up during general-purpose fine-tuning. Retrieval-Augmented Fine-Tuning (RAFT) (Zhang et al., 2024) closes this gap by fine-tuning the model specifically on the task of answering questions given a mix of relevant and irrelevant retrieved documents, so it explicitly learns to identify which parts of the context are useful and to ignore distractors, rather than hoping that behavior emerges from general instruction tuning.

How RAFT constructs its training data

For each training question, RAFT assembles a context window containing the oracle document (the one that actually contains the answer) mixed in with several distractor documents (plausible-looking but irrelevant retrievals) — mimicking what a real retriever will actually return at inference time, including its mistakes. The target output is a chain-of-thought answer that explicitly cites the relevant passage from the oracle document before stating the final answer. Critically, RAFT also trains on some examples where the oracle document is deliberately *excluded*, teaching the model to recognize when the provided context doesn't contain the answer and to say so rather than fabricate one.

raft_training_example_construction.py

python

import random

def build_raft_example(question, oracle_doc, distractor_pool, num_distractors=3, p_include_oracle=0.8):
    distractors = random.sample(distractor_pool, num_distractors)
    include_oracle = random.random() < p_include_oracle
    docs = distractors + ([oracle_doc] if include_oracle else [])
    random.shuffle(docs)

    if include_oracle:
        target = (
            f"According to the passage: '{oracle_doc['key_sentence']}', "
            f"the answer is {oracle_doc['answer']}."
        )
    else:
        target = "The provided documents do not contain enough information to answer this question."

    return {
        "input": {"question": question, "context_docs": docs},
        "target": target,
    }

# A RAFT dataset is built from many such examples across the target domain,
# with the oracle-inclusion rate tuned so the model learns both
# "extract the right answer" and "recognize when it's missing."

RAG vs fine-tuning vs RAFT

Approach	Learns to use retrieved context well?	Adapts to domain-specific knowledge?	Training cost
RAG with a general-purpose model	Only as well as general instruction tuning allows	No, relies entirely on retrieval at inference	None (inference-time only)
Fine-tuning without retrieval	N/A	Yes, but knowledge gets baked in and goes stale	Moderate-to-high
RAFT (fine-tuned specifically with retrieval-style context)	Yes, explicitly trained for it	Yes, plus stays current via retrieval at inference	Moderate-to-high, plus still needs a retriever

RAFT is closed-book fine-tuning plus open-book skill

A useful way to think about RAFT: it combines the domain adaptation benefit of standard supervised fine-tuning (the model gets better at the target domain's question style, vocabulary, and expected answer format) with explicit training on the open-book skill of filtering signal from noise in retrieved context — a skill that doesn't automatically transfer from general instruction tuning, especially for models that weren't heavily trained on long, noisy, multi-document contexts.

When it's worth the extra training investment

RAFT makes the most sense when you have a stable, narrow domain (internal documentation, a specific technical corpus, a legal or medical knowledge base) where off-the-shelf RAG with a general model shows a consistent pattern of being distracted by irrelevant retrieved chunks or failing to say 'I don't know' when the answer genuinely isn't in the retrieved context. It's a heavier investment than prompt engineering alone — it requires building a training set that resembles real retrieval noise and running an actual fine-tuning job — so it's typically reserved for cases where prompt-level fixes (reranking, better retrieval, stronger grounding instructions) have already been tried and plateaued.

The fine-tuned model is still coupled to a specific retriever's failure modes

RAFT trains the model against distractors sampled to resemble what your particular retriever actually returns. If you later swap retrievers, change the chunking strategy, or significantly expand the knowledge base, the distribution of 'typical noise' the model was trained to filter may no longer match reality, degrading the benefit until the fine-tuning data (and the fine-tune itself) is refreshed.

What's Next?

RAFT is one example of adapting a large pretrained model to a narrower task cheaply and durably — the broader family of techniques for doing this, especially when full fine-tuning is too expensive, is worth understanding as its own area.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →