mlsanityData to 7 Oct 2026

The Capacity for Moral Self-Correction in Large Language Models

arXiv 2302.07459 ↗First posted 15 Feb 2023cs.CLcited by 171

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

also in 1 more
Abstract (arXiv)

We test the hypothesis that language models trained with reinforcement learning from human feedback (RLHF) have the capability to "morally self-correct" -- to avoid producing harmful outputs -- if instructed to do so. We find strong evidence in support of this hypothesis across three different experiments, each of which reveal different facets of moral self-correction. We find that the capability for moral self-correction emerges at 22B model parameters, and typically improves with increasing model size and RLHF training. We believe that at this level of scale, language models obtain two capabilities that they can use for moral self-correction: (1) they can follow instructions and (2) they can learn complex normative concepts of harm like stereotyping, bias, and discrimination. As such, they can follow instructions to avoid certain kinds of morally harmful outputs. We believe our results are cause for cautious optimism regarding the ability to train language models to abide by ethical principles.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 71 references that are in the map. First posted on arXiv 15 Feb 2023.

Builds on

46 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  2. 2020Learning to summarize from human feedback2 Sep

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
32 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  4. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  6. 2017Counterfactual Fairnesscited by 802
  7. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  8. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  9. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  10. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  12. 2018Gender Bias in Coreference Resolutioncited by 438
  13. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  14. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  15. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  16. 2019Measuring Bias in Contextualized Word Representationscited by 275
  17. 2022Predictability and Surprise in Large Generative Modelscited by 209
  18. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  19. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  20. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  21. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  22. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  23. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 96
  24. 2019Evaluating the Underlying Gender Bias in Contextualized Word Embeddingscited by 94
  25. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  26. 2022Inverse scaling can become U-shapedcited by 76
  27. 2019Counterfactual Risk Assessments, Evaluation, and Fairnesscited by 59
  28. 2021The Use and Misuse of Counterfactuals in Ethical Machine Learningcited by 52
  29. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
  30. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  31. 2020Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis?cited by 27
  32. 2022Counterfactual Fairness Is Basically Demographic Paritycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 56; 2024: 61; 2025: 36; 2026: 18.

171 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  4. 2023Teaching Large Language Models to Self-Debugcited by 963
  5. 2023Large Language Models as Optimizerscited by 830
  6. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  7. 2023A Comprehensive Overview of Large Language Modelscited by 639
  8. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  9. 2023Cognitive Architectures for Language Agentscited by 422
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  11. 2023Language Models can Solve Computer Taskscited by 371
  12. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
38 more
  1. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  2. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  3. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  4. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  5. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  6. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  7. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  8. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  9. 2023Self-Alignment with Instruction Backtranslationcited by 149
  10. 2024LLM Critics Help Catch LLM Bugscited by 138
  11. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  12. 2023Generative AIcited by 119
  13. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 117
  14. 2024Critique-out-Loud Reward Modelscited by 105
  15. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  16. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  17. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  18. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  19. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  20. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  21. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
  22. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  23. 2023Eight Things to Know about Large Language Modelscited by 73
  24. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  25. 2023Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4cited by 67
  26. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 64
  27. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 61
  28. 2025Contextual Agent Security: A Policy for Every Purposecited by 55
  29. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  30. 2023Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Valuescited by 49
  31. 2024Evaluating Gender Bias in Large Language Models via Chain-of-Thought Promptingcited by 49
  32. 2024Measuring Implicit Bias in Explicitly Unbiased Large Language Modelscited by 49
  33. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  34. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  35. 2023Self-Detoxifying Language Models via Toxification Reversalcited by 45
  36. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  37. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  38. 2023Specific versus General Principles for Constitutional AIcited by 41

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-correction is Not An Innate Capability in Language Models
  2. 2026Training Large Language Models for Self-Explanation Faithfulness
  3. 2026Do VLMs Have a Moral Backbone? A Study on the Fragile Morality of Vision-Language Models
  4. 2024Dishonesty in Helpful and Harmless Alignment
  5. 2026The Fragility Of Moral Judgment In Large Language Models
  6. 2026CORDA: A Benchmark for Hierarchical Harm-Centric Moral Reasoning in Large Language Models
  7. 2026Unsupervised Elicitation of Moral Values from Language Models
  8. 2025Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework