Constitutional AI
Training a model against a written set of principles instead of human ratings — which scales, and moves the question from "what did raters prefer" to "who wrote the principles."
When not to use it
- With vague principles. "Be helpful" isn't a critique the model can perform. Specificity is the whole requirement.
- When the critique model is weak. It can't supervise what it can't evaluate.
- As an escape from value choices. It makes them explicit; it doesn't remove them.
- Assuming human preferences are gone. The critique model was trained on human feedback. One step removed, not absent.
Reach for something else instead
- RLHF — human raters, expensive, unwritten values, and a ceiling at human ability.
- DPO on human preferences — simpler, same ceiling.
- Debate — models arguing, a human judging. Another scalable-oversight attempt.
- Expert raters — works, doesn't scale, and it's the thing this is trying to replace.
Sources & further reading
- Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback — the method; harmlessness training with no human harmlessness labels.
- Lee et al. (2023), RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — AI feedback matching human feedback across tasks.
- Irving, Christiano & Amodei (2018), AI Safety via Debate — the scalable-oversight problem this all belongs to.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading it as removing human values. It relocates them into a document and an author.
- Writing principles too vague to act on. The model has to be able to perform the critique.
- Ignoring principle conflicts. Helpful and harmless collide; if the constitution doesn't resolve it, the model will.
- Missing why the authorship question gets asked here. It's because the values are finally visible.
At a glance
FieldSafety & Ethics
What it replaceshuman raters, with a written document
Two stagesself-critique and revision, then RL on AI-generated preferences
Why it worksevaluating against a principle is easier than generating compliance
The real gainthe values are written down
The open questionwho writes them
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Constitutional AI vs. RLHF — one writes its values in a document you can read and argue with; the other leaves them as an unwritten average of what contractors happened to prefer.