mlsanityData to 14 Oct 2026

Specific versus General Principles for Constitutional AI

arXiv 2310.13798 ↗First posted 20 Oct 2023cs.CL cs.AIcited by 47

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Human feedback can prevent overtly harmful utterances in conversational models, but may not automatically mitigate subtle problematic behaviors such as a stated desire for self-preservation or power. Constitutional AI offers an alternative, replacing human feedback with feedback from AI models conditioned only on a list of written principles. We find this approach effectively prevents the expression of such behaviors. The success of simple principles motivates us to ask: can models learn general ethical behaviors from only a single written principle? To test this, we run experiments using a principle roughly stated as "do what's best for humanity". We find that the largest dialogue models can generalize from this short constitution, resulting in harmless assistants with no stated interest in specific motivations like power. A general principle may thus partially avoid the need for a long list of constitutions targeting potentially harmful behaviors. However, more detailed constitutions still improve fine-grained control over specific types of harms. This suggests both general and specific principles have value for steering AI safely.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 42 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

23 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023The Capacity for Moral Self-Correction in Large Language Models15 Feb
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels28 Oct

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2022Red Teaming Language Models with Language Modelscited by 1,026
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  12. 2022Large Language Models Can Self-Improvecited by 664
7 more
  1. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  2. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  3. 2022Self-critiquing models for assisting human evaluatorscited by 380
  4. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  5. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  6. 2022Learning by Distilling Contextcited by 138
  7. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 14; 2026: 14.

47 papers.

Sort:

By area:

  1. 2024Alignment faking in large language modelscited by 360
  2. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  4. 2024Generative Reward Modelscited by 148
  5. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  6. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  7. 2024Rule Based Rewards for Language Model Safetycited by 108
  8. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  9. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  10. 2024A Language Model's Guide Through Latent Spacecited by 43
  11. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 41
  12. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
35 more
  1. 2025Generating Robot Constitutions & Benchmarks for Semantic Safetycited by 33
  2. 2024Standards for Belief Representations in LLMscited by 30
  3. 2025QA-LIGN: Aligning LLMs through Constitutionally Decomposed QAcited by 28
  4. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
  5. 2025Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIcited by 24
  6. 2026Model Spec Midtraining: Improving How Alignment Training Generalizescited by 20
  7. 2024Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labelscited by 19
  8. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 15
  9. 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 12
  10. 2024LLMCRIT: Teaching Large Language Models to Use Criteriacited by 9
  11. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8
  12. 2024Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarioscited by 7
  13. 2025Data-adaptive Safety Rules for Training Reward Modelscited by 6
  14. 2024Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviorscited by 5
  15. 2025ENCORE: Entropy-guided Reward Composition for Multi-head Safety Reward Modelscited by 4
  16. 2025Domain-Specific Constitutional AI: Enhancing Safety in LLM-Powered Mental Health Chatbotscited by 4
  17. 2026Contextual Safety Reasoning and Grounding for Open-World Robotscited by 4
  18. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Modelscited by 4
  19. 2024Aligning Large Language Models from Self-Reference AI Feedback with one General Principlecited by 2
  20. 2026Human Society-Inspired Approaches to Agentic AI Security: The 4C Frameworkcited by 2
  21. 2026LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behaviorcited by 2
  22. 2024Supertrust foundational alignment: mutual trust must replace permanent control for safe superintelligencecited by 1
  23. 2024Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignmentcited by 1
  24. 2025Contemplative Artificial Intelligencecited by 1
  25. 2026Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modelingcited by 1
  26. 2026ATLAS: Constitution-Conditioned Latent Geometry and Redistribution Across Language Models and Neural Perturbation Datacited by 1
  27. 2026Emergent alignment and the projectability of ethical personascited by 1
  28. 2026Constitutional Value Potentials: reading and steering internal priority margins in language modelscited by 1
  29. 2025A Peek Behind the Curtain: Using Step-Around Prompt Engineering to Identify Bias and Misinformation in GenAI Modelscited by 0
  30. 2025Ethics Readiness of Artificial Intelligence: A Practical Evaluation Methodcited by 0
  31. 2026Ethical Risks in Deploying Large Language Models: An Evaluation of Medical Ethics Jailbreakingcited by 0
  32. 2026Sustaining AI safety: Control-theoretic external impossibility, intrinsic necessity, and structural requirementscited by 0
  33. 2026A Virtuous AI is an Existential Riskcited by 0
  34. 2026AgentPersonaBench: Benchmarking Persona-Driven User Simulationcited by 0
  35. 2026SIGMA: Self-Improving Alignment Generalization from a Model Speccited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values
  2. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback
  3. 2024Should agentic conversational AI change how we think about ethics? Characterising an interactional ethics centred on respect
  4. 2024Informed AI Regulation: Comparing the Ethical Frameworks of Leading LLM Chatbots Using an Ethics-Based Audit to Assess Moral Reasoning and Normative Values
  5. 2024Suppressing Pink Elephants with Direct Principle Feedback
  6. 2026Statutory AI: Aligning Large Language Models With Legal Norms
  7. 2025Auditing the Ethical Logic of Generative AI Models
  8. 2025Normative Conflicts and Shallow AI Alignment