Procedural Dilemma Generation for Evaluating Moral Reasoning in Humans and Language Models
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
As AI systems like language models are increasingly integrated into decision-making processes affecting people's lives, it's critical to ensure that these systems have sound moral reasoning. To test whether they do, we need to develop systematic evaluations. We provide a framework that uses a language model to translate causal graphs that capture key aspects of moral dilemmas into prompt templates. With this framework, we procedurally generated a large and diverse set of moral dilemmas -- the OffTheRails benchmark -- consisting of 50 scenarios and 400 unique test items. We collected moral permissibility and intention judgments from human participants for a subset of our items and compared these judgments to those from two language models (GPT-4 and Claude-2) across eight conditions. We find that moral dilemmas in which the harm is a necessary means (as compared to a side effect) resulted in lower permissibility and higher intention ratings for both participants and language models. The same pattern was observed for evitable versus inevitable harmful outcomes. However, there was no clear effect of whether the harm resulted from an agent's action versus from having omitted to act. We discuss limitations of our prompt generation pipeline and opportunities for improving scenarios to increase the strength of experimental effects.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 94 references that are in the map. First posted on arXiv 17 Apr 2024.
Builds on
12 of 94 references.
Closest ancestors · 2
Same direction, newest first
- 2023Exploring the psychology of LLMs' Moral and Legal Reasoning2 Aug
- 2023Understanding Social Reasoning in Language Models with Language Models21 Jun
Other references · 10
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
- 2021Can Machines Learn Morality? The Delphi Experimentcited by 140
- 2020Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequencescited by 133
- 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
- 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
- 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
- 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
- 2023Social Contract AI: Aligning AI Assistants with Implicit Group Normscited by 9
Built on by
Citing papers per year: 2024: 2; 2025: 0; 2026: 2.
4 papers.
- 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
- 2024Human-like Affective Cognition in Foundation Modelscited by 13
- 2026Visuospatial Perspective Taking in Multimodal Language Modelscited by 2
- 2026Are LLMs Bad at Moral Reasoning?cited by 0
- 2026Are LLMs Bad at Moral Reasoning?10 Jun 2026
- 2026Visuospatial Perspective Taking in Multimodal Language Models4 Mar 2026
- 2024Human-like Affective Cognition in Foundation Models18 Sep 2024
- 2024Language Model Alignment in Multilingual Trolley Problems2 Jul 2024
Similar, not cited
Close in wording, no citation either way
- 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes
- 2026CORDA: A Benchmark for Hierarchical Harm-Centric Moral Reasoning in Large Language Models
- 2025The Morality of Probability: How Implicit Moral Biases in LLMs May Shape the Future of Human-AI Symbiosis
- 2025"Pull or Not to Pull?'': Investigating Moral Biases in Leading Large Language Models Across Ethical Dilemmas
- 2025One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning
- 2025The Convergent Ethics of AI? Analyzing Moral Foundation Priorities in Large Language Models with a Multi-Framework Approach
- 2026Reasoning or Rhetoric? An Empirical Analysis of Moral Reasoning Explanations in Large Language Models
- 2024The Moral Turing Test: Evaluating Human-LLM Alignment in Moral Decision-Making