Constitutional AI & Alignment Techniques

The limits of pure RLHF

Reinforcement Learning from Human Feedback (RLHF) aligns a model's behavior by having humans compare pairs of model outputs and say which they prefer, training a reward model on those preferences, and then fine-tuning the LLM to produce outputs the reward model scores highly. This works well, but it has real limits: it requires large volumes of human labeling, which is slow and expensive; human raters can disagree, get fatigued, or hold inconsistent standards; and the model's notion of 'good behavior' is only as clear as thousands of individual pairwise judgments can make it — there's no single explicit statement of what the model is actually supposed to value.

Constitutional AI was developed to address this by giving the model an explicit, written set of principles — a 'constitution' — and training it to evaluate and revise its own outputs against those principles, reducing (though not eliminating) the dependence on human-labeled preference data for every single judgment.

The core loop: self-critique and revision

The constitutional AI process generally has two stages that build on each other:

Supervised stage (self-critique and revision): The model is prompted to generate a response to some input, then prompted again to critique its own response against a specific principle from the constitution ('Did this response contain anything discriminatory or unsafe?'), and then prompted to revise the response to fix whatever the critique found. This critique-and-revise loop can run through multiple principles and multiple rounds. The resulting revised responses — produced by the model itself, with no human writing new examples — become a training set for supervised fine-tuning.

Reinforcement learning stage (RLAIF): Instead of (or in addition to) human-labeled preference pairs, the model is shown two of its own outputs and asked to judge which better satisfies the constitution. These AI-generated preference judgments train a reward model, which is then used for reinforcement learning, much like in RLHF — except the preference labels come from the model referencing explicit principles rather than solely from human raters. This step is often called RLAIF (Reinforcement Learning from AI Feedback).

RLHF vs Constitutional AI / RLAIF

Complementary, not a replacement

In practice, labs don't choose one or the other — constitutional AI methods and human feedback are typically combined. Humans still write the constitution itself, still provide ground-truth preference data for behaviors that are hard to specify as explicit principles (tone, helpfulness, nuanced judgment calls), and still audit and red-team the resulting model's behavior. Constitutional methods reduce how much human labeling is needed for applying a given standard at scale, but the standard itself, and the oversight of whether the whole system actually meets it, still rests on human judgment.

This is also why 'alignment' in modern LLM training is usually a multi-stage pipeline: pretraining on broad data, supervised fine-tuning on curated examples (including constitutional self-revisions), and reinforcement learning (from human feedback, AI feedback, or both) layered on top.

Why this matters for agentic systems

As models take on more autonomous, multi-step tasks — calling tools, taking real-world actions, operating with less direct human supervision per step — the question of what principles govern their behavior in situations nobody explicitly anticipated becomes more important, not less. A model trained to reason about explicit principles, rather than one that only pattern-matches to labeled examples, is generally expected to generalize alignment behavior more robustly to novel situations than one trained purely on a fixed set of preference comparisons, though this remains an active area of ongoing evaluation rather than a solved problem.

What's next

Constitutional AI shapes what a model is willing to do at the training level. But even a well-aligned model still needs a reliable way to actually use tools and take actions in agentic systems. The next lesson covers tool-use patterns in LLM agents — how function-calling schemas work and how a model decides which tool to invoke.