# Specific versus General Principles for Constitutional AI

arXiv: https://arxiv.org/abs/2310.13798 · HTML: https://mlsanity.com/paper/2310.13798/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [The Capacity for Moral Self-Correction in Large Language Models](https://mlsanity.com/paper/2302.07459/) (2023)
- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?](https://mlsanity.com/paper/2106.01465/) (2021)
- [When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels](https://mlsanity.com/paper/2210.15893/) (2022)

## Built on by (most cited first, 30 of 47)

- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [Secrets of RLHF in Large Language Models Part II: Reward Modeling](https://mlsanity.com/paper/2401.06080/) (2024)
- [Generative Reward Models](https://mlsanity.com/paper/2410.12832/) (2024)
- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025)
- [Rule Based Rewards for Language Model Safety](https://mlsanity.com/paper/2411.01111/) (2024)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It](https://mlsanity.com/paper/2505.24119/) (2025)
- [A Language Model's Guide Through Latent Space](https://mlsanity.com/paper/2402.14433/) (2024)
- [Inverse Constitutional AI: Compressing Preferences into Principles](https://mlsanity.com/paper/2406.06560/) (2024)
- [Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.16534/) (2025)
- [Generating Robot Constitutions & Benchmarks for Semantic Safety](https://mlsanity.com/paper/2503.08663/) (2025)
- [Standards for Belief Representations in LLMs](https://mlsanity.com/paper/2405.21030/) (2024)
- [QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA](https://mlsanity.com/paper/2506.08123/) (2025)
- [Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback](https://mlsanity.com/paper/2401.11458/) (2024)
- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [Model Spec Midtraining: Improving How Alignment Training Generalizes](https://mlsanity.com/paper/2605.02087/) (2026)
- [Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels](https://mlsanity.com/paper/2404.14313/) (2024)
- [Pre-Trained Policy Discriminators are General Reward Models](https://mlsanity.com/paper/2507.05197/) (2025)
- [RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Data](https://mlsanity.com/paper/2505.19030/) (2025)
- [LLMCRIT: Teaching Large Language Models to Use Criteria](https://mlsanity.com/paper/2403.01069/) (2024)
- [POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization](https://mlsanity.com/paper/2410.12999/) (2024)
- [Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarios](https://mlsanity.com/paper/2407.19760/) (2024)
- [Data-adaptive Safety Rules for Training Reward Models](https://mlsanity.com/paper/2501.15453/) (2025)
- [Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors](https://mlsanity.com/paper/2409.18203/) (2024)
- [ENCORE: Entropy-guided Reward Composition for Multi-head Safety Reward Models](https://mlsanity.com/paper/2503.20995/) (2025)
- [Domain-Specific Constitutional AI: Enhancing Safety in LLM-Powered Mental Health Chatbots](https://mlsanity.com/paper/2509.16444/) (2025)
- [Contextual Safety Reasoning and Grounding for Open-World Robots](https://mlsanity.com/paper/2602.19983/) (2026)
- [Reinforcement Learning Towards Broadly and Persistently Beneficial Models](https://mlsanity.com/paper/2606.24014/) (2026)

## Abstract (arXiv)

Human feedback can prevent overtly harmful utterances in conversational models, but may not automatically mitigate subtle problematic behaviors such as a stated desire for self-preservation or power. Constitutional AI offers an alternative, replacing human feedback with feedback from AI models conditioned only on a list of written principles. We find this approach effectively prevents the expression of such behaviors. The success of simple principles motivates us to ask: can models learn general ethical behaviors from only a single written principle? To test this, we run experiments using a principle roughly stated as "do what's best for humanity". We find that the largest dialogue models can generalize from this short constitution, resulting in harmless assistants with no stated interest in specific motivations like power. A general principle may thus partially avoid the need for a long list of constitutions targeting potentially harmful behaviors. However, more detailed constitutions still improve fine-grained control over specific types of harms. This suggests both general and specific principles have value for steering AI safely.