mlsanityData to 14 Oct 2026

Evaluating the Moral Beliefs Encoded in LLMs

arXiv 2307.14324 ↗First posted 26 Jul 2023cs.CL cs.AI cs.CYcited by 215

Not on the map. Why?

Cited by 24 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a case study on the design, administration, post-processing, and evaluation of surveys on large language models (LLMs). It comprises two components: (1) A statistical method for eliciting beliefs encoded in LLMs. We introduce statistical measures and evaluation metrics that quantify the probability of an LLM "making a choice", the associated uncertainty, and the consistency of that choice. (2) We apply this method to study what moral beliefs are encoded in different LLMs, especially in ambiguous cases where the right choice is not obvious. We design a large-scale survey comprising 680 high-ambiguity moral scenarios (e.g., "Should I tell a white lie?") and 687 low-ambiguity moral scenarios (e.g., "Should I stop for a pedestrian on the road?"). Each scenario includes a description, two possible actions, and auxiliary labels indicating violated rules (e.g., "do not kill"). We administer the survey to 28 open- and closed-source LLMs. We find that (a) in unambiguous scenarios, most models "choose" actions that align with commonsense. In ambiguous cases, most models express uncertainty. (b) Some models are uncertain about choosing the commonsense action because their responses are sensitive to the question-wording. (c) Some models reflect clear preferences in ambiguous scenarios. Specifically, closed-source models tend to agree with each other.

Key facts. Cited by 215 papers in the map (data through 14 Oct 2026). Builds on 45 of its 76 references that are in the map. First posted on arXiv 26 Jul 2023.

Builds on

45 of 76 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2016Concrete Problems in AI Safetycited by 1,642
33 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  2. 2023PaLM 2 Technical Reportcited by 1,162
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  4. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  5. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  6. 2023Whose Opinions Do Language Models Reflect?cited by 773
  7. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  10. 2020Aligning AI With Shared Human Valuescited by 723
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  12. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  13. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  14. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
  15. 2021Unsolved Problems in ML Safetycited by 292
  16. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  17. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  18. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  19. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  20. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  21. 2022Talking About Large Language Modelscited by 175
  22. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  23. 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
  24. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 135
  25. 2022Language Models as Agent Modelscited by 132
  26. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 101
  27. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  28. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  29. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  30. 2022Meaning without reference in large language modelscited by 76
  31. 2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 74
  32. 2022Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophycited by 31
  33. 2022An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 64; 2025: 71; 2026: 69.

215 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  3. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  4. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  5. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  6. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  7. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  8. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  9. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  10. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  11. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  12. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
38 more
  1. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  2. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  3. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  4. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 59
  5. 2024Large Language Model Safety: A Holistic Surveycited by 59
  6. 2023On the steerability of large language models toward data-driven personascited by 53
  7. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 52
  8. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 47
  9. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  10. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  11. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 43
  12. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 37
  13. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 29
  14. 2024A Survey of Useful LLM Evaluationcited by 29
  15. 2023Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBenchcited by 28
  16. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 28
  17. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 28
  18. 2024Evaluating language models as risk scorescited by 27
  19. 2024Strong and weak alignment of large language models with human valuescited by 27
  20. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  21. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  22. 2025Normative Evaluation of Large Language Models with Everyday Moral Dilemmascited by 24
  23. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Modelscited by 23
  24. 2024MoralBERT: A Fine-Tuned Language Model for Capturing Moral Values in Social Discussionscited by 23
  25. 2024Can We Talk Models Into Seeing the World Differently?cited by 23
  26. 2024LLMs Are Prone to Fallacies in Causal Inferencecited by 23
  27. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  28. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  29. 2024Evaluating Large Language Models with Psychometricscited by 22
  30. 2025Multiple LLM Agents Debate for Equitable Cultural Alignmentcited by 21
  31. 2025AI Alignment at Your Discretioncited by 20
  32. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scalecited by 20
  33. 2024ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Modelscited by 19
  34. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  35. 2024The 2024 Foundation Model Transparency Indexcited by 19
  36. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  37. 2025Unmasking Conversational Bias in AI Multiagent Systemscited by 17
  38. 2024CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Modelscited by 16

50 most cited of 215 All 215 (JSON)