mlsanityData to 7 Oct 2026

When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment

arXiv 2210.01478 ↗First posted 4 Oct 2022cs.CL cs.AI cs.CYcited by 93

Not on the map. Why?

Abstract (arXiv)

AI systems are becoming increasingly intertwined with human life. In order to effectively collaborate with humans and ensure safety, AI systems need to be able to understand, interpret and predict human moral judgments and decisions. Human moral judgments are often guided by rules, but not always. A central challenge for AI safety is capturing the flexibility of the human moral mind -- the ability to determine when a rule should be broken, especially in novel or unusual situations. In this paper, we present a novel challenge set consisting of rule-breaking question answering (RBQA) of cases that involve potentially permissible rule-breaking -- inspired by recent moral psychology studies. Using a state-of-the-art large language model (LLM) as a basis, we propose a novel moral chain of thought (MORALCOT) prompting strategy that combines the strengths of LLMs with theories of moral reasoning developed in cognitive science to predict human moral judgments. MORALCOT outperforms seven existing LLMs by 6.2% F1, suggesting that modeling human reasoning might be necessary to capture the flexibility of the human moral mind. We also conduct a detailed error analysis to suggest directions for future work to improve AI safety using RBQA. Our data is open-sourced at https://huggingface.co/datasets/feradauto/MoralExceptQA and code at https://github.com/feradauto/MoralCoT

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 39 of its 84 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

39 of 84 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
27 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2019Language Models as Knowledge Bases?cited by 1,922
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  13. 2020Aligning AI With Shared Human Valuescited by 706
  14. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  15. 2019ELI5: Long Form Question Answeringcited by 567
  16. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  17. 2018AI safety via debatecited by 384
  18. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  19. 2021Unsolved Problems in ML Safetycited by 291
  20. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  21. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  22. 2021Alignment of Language Agentscited by 167
  23. 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 133
  24. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 100
  25. 2018Conversational AI: The Science Behind the Alexa Prizecited by 63
  26. 2020Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledgecited by 24
  27. 2022When Is It Acceptable to Break the Rules? Knowledge Representation of Moral Judgement Based on Empirical Datacited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 28; 2024: 24; 2025: 20; 2026: 20.

93 papers.

Sort:

By area:

  1. 2023Cognitive Architectures for Language Agentscited by 422
  2. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  3. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 211
  4. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
  5. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  6. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  7. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  8. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  9. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  10. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 57
  11. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  12. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
38 more
  1. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  2. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  3. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  4. 2024Beyond Preferences in AI Alignmentcited by 46
  5. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  6. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  7. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergencecited by 42
  8. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  9. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 30
  10. 2024AI for social science and social science of AI: A Surveycited by 29
  11. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  12. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  13. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  14. 2023Susceptibility to Influence of Large Language Modelscited by 17
  15. 2023Interpretable Math Word Problem Solution Generation Via Step-by-step Planningcited by 17
  16. 2023What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situationscited by 15
  17. 2025Are Rules Meant to be Broken? Understanding Multilingual Moral Reasoning as a Computational Pipeline with UniMoralcited by 14
  18. 2025When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmascited by 14
  19. 2023Values, Ethics, Morals? On the Use of Moral Concepts in NLP Researchcited by 13
  20. 2025The Staircase of Ethics: Probing LLM Value Priorities through Multi-Step Induction to Complex Moral Dilemmascited by 13
  21. 2024Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Futurecited by 10
  22. 2025Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMscited by 10
  23. 2023"Oops, Did I Just Say That?" Testing and Repairing Unethical Suggestions of Large Language Models with Suggest-Critique-Reflect Processcited by 9
  24. 2023Toward Grounded Commonsense Reasoningcited by 9
  25. 2024CMoralEval: A Moral Evaluation Benchmark for Chinese Large Language Modelscited by 9
  26. 2024Recent Advancement of Emotion Cognition in Large Language Modelscited by 9
  27. 2024MM-MoralBench: A MultiModal Moral Evaluation Benchmark for Large Vision-Language Modelscited by 9
  28. 2025Can AI Model the Complexities of Human Moral Decision-Making? A Qualitative Study of Kidney Allocation Decisionscited by 9
  29. 2022Revision Transformers: Instructing Language Models to Change their Valuescited by 8
  30. 2023Towards Theory-based Moral AI: Moral AI with Aggregating Models Based on Normative Ethical Theorycited by 8
  31. 2023Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Normscited by 8
  32. 2023Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Modelscited by 7
  33. 2024Morality is Non-Binary: Building a Pluralist Moral Sentence Embedding Space using Contrastive Learningcited by 7
  34. 2023Minimum Levels of Interpretability for Artificial Moral Agentscited by 5
  35. 2024Integration of cognitive tasks into artificial general intelligence test for large modelscited by 5
  36. 2024GPT-ology, Computational Models, Silicon Sampling: How should we think about LLMs in Cognitive Science?cited by 5
  37. 2025SPRI: Aligning Large Language Models with Context-Situated Principlescited by 5
  38. 2023ValueDCG: Measuring Comprehensive Human Value Understanding Ability of Language Modelscited by 4

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes
  2. 2024The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making
  3. 2025Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework
  4. 2026CORDA: A Benchmark for Hierarchical Harm-Centric Moral Reasoning in Large Language Models
  5. 2026Are LLMs Bad at Moral Reasoning?
  6. 2025The Convergent Ethics of AI? Analyzing Moral Foundation Priorities in Large Language Models with a Multi-Framework Approach
  7. 2026Beyond Binary Moral Judgment: Modeling Ethical Pluralism in AI
  8. 2026Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picture