mlsanityData to 7 Oct 2026

Constitutional AI: Harmlessness from AI Feedback

arXiv 2212.08073 ↗First posted 15 Dec 2022cs.CL cs.AIcited by 2,950

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.

Key facts. Cited by 2,950 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 82 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

26 of 82 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2022Red Teaming Language Models with Language Modelscited by 983
14 more
  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  2. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2022Large Language Models Can Self-Improvecited by 656
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2018AI safety via debatecited by 384
  7. 2022Self-critiquing models for assisting human evaluatorscited by 372
  8. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  9. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  10. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  11. 2018Supervising strong learners by amplifying weak expertscited by 139
  12. 2022Training Language Models with Language Feedbackcited by 48
  13. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  14. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 400; 2024: 720; 2025: 813; 2026: 1015.

2,950 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  12. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
38 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  5. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  6. 2023LIMA: Less Is More for Alignmentcited by 1,080
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  9. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  10. 2023Teaching Large Language Models to Self-Debugcited by 963
  11. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  12. 2023Large Language Models as Optimizerscited by 830
  13. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  14. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  15. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  16. 2023Whose Opinions Do Language Models Reflect?cited by 759
  17. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  18. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  19. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  20. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  21. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  22. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  23. 2024DeepSeek-V3 Technical Reportcited by 724
  24. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  25. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  26. 2023Instruction Tuning with GPT-4cited by 676
  27. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  28. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  29. 2024Self-Rewarding Language Modelscited by 654
  30. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  31. 2024A Survey on Large Language Models for Code Generationcited by 646
  32. 2023A Comprehensive Overview of Large Language Modelscited by 639
  33. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  34. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  35. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  36. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  37. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  38. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552

50 most cited of 2,950 All 2,950 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Constitution or Collapse? Exploring Constitutional AI with Llama 3-8B
  2. 2025Decoding Human Preferences in Alignment: An Improved Approach to Inverse Constitutional AI
  3. 2026Why Does RLAIF Work At All?
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback
  5. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Models
  6. 2024Evaluating Defences against Unsafe Feedback in RLHF
  7. 2025Teaching AI to Handle Exceptions: Supervised Fine-Tuning with Human-Aligned Judgment
  8. 2026Subliminal Signals in Preference Labels