How Utilitarian Are OpenAI's Models Really? Replicating and Reinterpreting Pfeffer, Kr"ugel, and Uhl (2025)
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Pfeffer, Kr"ugel, and Uhl (2025) report that OpenAI's reasoning model o1-mini produces more utilitarian responses to the trolley problem and footbridge dilemma than the non-reasoning model GPT-4o, and they raise the question whether growing reasoning capabilities bring about a "utilitarian turn" in LLMs. I extend their exploratory study in a direction they call for: with four current OpenAI models and systematic prompt variation. On the trolley dilemma, the hypothesized utilitarian turn is not confirmed. GPT-4o's low utilitarian rate reflects safety refusals triggered by the prompt's advisory framing rather than a deontological commitment; on reformulated prompt variants -- for instance, agent-neutral "Is it morally permissible...?" instead of advisory "Should I...?" -- all four models, reasoning or not, converge on utilitarian answers. The footbridge finding is partially confirmed: reasoning models tend to give more utilitarian responses than non-reasoning models across prompt variations, but they often refuse to answer or answer non-utilitarian. These results demonstrate that single-prompt evaluations of LLM moral responses are unreliable: multi-prompt robustness testing should be standard practice for any empirical claims about LLM behavior.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 7 of its 14 references that are in the map. First posted on arXiv 24 Mar 2026.
Builds on
7 of 14 references.
References · 7
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
- 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
- 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
- 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
- 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
- 2025From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychologycited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Incoherent by Design? On the Moral Self-Consistency of LLMs
- 2026Are LLMs Bad at Moral Reasoning?
- 2026Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMs
- 2025"Pull or Not to Pull?'': Investigating Moral Biases in Leading Large Language Models Across Ethical Dilemmas
- 2026Measuring AI Accountability Through Argumentation Analysis: Can Model Reasoning Withstand Scrutiny?
- 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes
- 2024Language Model Alignment in Multilingual Trolley Problems
- 2024A dataset of questions on decision-theoretic reasoning in Newcomb-like problems