# The Greatest Good Benchmark: Measuring LLMs' Alignment with Utilitarian Moral Dilemmas

arXiv: https://arxiv.org/abs/2503.19598 · HTML: https://mlsanity.com/paper/2503.19598/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 13 of 13)

- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)
- [CogBench: a large language model walks into a psychology lab](https://mlsanity.com/paper/2402.18225/) (2024)
- [Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations](https://mlsanity.com/paper/2305.13299/) (2023)
- [Values, Ethics, Morals? On the Use of Moral Concepts in NLP Research](https://mlsanity.com/paper/2310.13915/) (2023)
- [Do Moral Judgment and Reasoning Capability of LLMs Change with Language? A Study using the Multilingual Defining Issues Test](https://mlsanity.com/paper/2402.02135/) (2024)
- [Towards Theory-based Moral AI: Moral AI with Aggregating Models Based on Normative Ethical Theory](https://mlsanity.com/paper/2306.11432/) (2023)
- [How should the advent of large language models affect the practice of science?](https://mlsanity.com/paper/2312.03759/) (2023)

## Built on by (most cited first, 12 of 12)

- [MAEBE: Multi-Agent Emergent Behavior Framework](https://mlsanity.com/paper/2506.03053/) (2025)
- [Many LLMs Are More Utilitarian Than One](https://mlsanity.com/paper/2507.00814/) (2025)
- [Exploring Cultural Variations in Moral Judgments with Large Language Models](https://mlsanity.com/paper/2506.12433/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [Are Language Models Sensitive to Morally Irrelevant Distractors?](https://mlsanity.com/paper/2602.09416/) (2026)
- [Structuring the Space of Sociotechnical Alignment](https://mlsanity.com/paper/2607.01250/) (2026)
- [GAIN: A Benchmark for Goal-Aligned Decision-Making of Large Language Models under Imperfect Norms](https://mlsanity.com/paper/2603.18469/) (2026)
- [The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment](https://mlsanity.com/paper/2607.05552/) (2026)
- [Theory Trace Card: Theory-Driven Socio-Cognitive Evaluation of LLMs](https://mlsanity.com/paper/2601.01878/) (2026)
- [Probing Ethical Framework Representations in Large Language Models: Structure, Entanglement, and Methodological Challenges](https://mlsanity.com/paper/2603.23659/) (2026)
- [Staying with the Uncertainty: Uncertainty-Scaffolding Strategies for Artificial Moral Advisors in LLM-to-LLM Simulated Conversations](https://mlsanity.com/paper/2606.05890/) (2026)
- [Are LLMs Bad at Moral Reasoning?](https://mlsanity.com/paper/2606.11635/) (2026)

## Abstract (arXiv)

The question of how to make decisions that maximise the well-being of all persons is very relevant to design language models that are beneficial to humanity and free from harm. We introduce the Greatest Good Benchmark to evaluate the moral judgments of LLMs using utilitarian dilemmas. Our analysis across 15 diverse LLMs reveals consistently encoded moral preferences that diverge from established moral theories and lay population moral standards. Most LLMs have a marked preference for impartial beneficence and rejection of instrumental harm. These findings showcase the 'artificial moral compass' of LLMs, offering insights into their moral alignment.