Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picture
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Recent work on evaluating the moral competence of large language models (LLMs) has focused primarily on what we call the moral value problem, i.e., whether model outputs align with human moral values. In contrast, the moral norm problem, i.e., whether models can identify and correctly apply context-sensitive moral norms, remains underexplored. We posit that this imbalance stems from the field's reliance on descriptive ethics frameworks, such as Moral Foundations Theory and Kohlberg's stages of moral development, which emphasize value representation over normative application. We review existing benchmarks and evaluation methods, and show that they cluster heavily around the value problem, while discussion regarding normative ethics remains underrepresented. We identify three crucial gaps: (i) the absence of high-quality ground-truth data for moral norms and their applications, (ii) insufficient evaluation of intermediate reasoning processes, and (iii) limited attention to the identification of morally relevant features in context. Subsequently, we propose a research agenda that includes the development of standardized formal representations for normative theories, the construction of expert-annotated datasets capturing norm application, and evaluation protocols that explicitly distinguish between values-level and norms-level competence. Our goal is to encourage a more systematic study of normative reasoning in LLMs.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 11 of its 16 references that are in the map. First posted on arXiv 29 May 2026.
Builds on
11 of 16 references.
Closest ancestors · 3
Same direction, newest first
- 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes18 Oct
- 2025Are Language Models Consequentialist or Deontological Moral Reasoners?27 May
- 2023Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?29 Aug
Other references · 8
- 2020Aligning AI With Shared Human Valuescited by 706
- 2023Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMscited by 64
- 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
- 2024Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundationscited by 19
- 2025Generative Value Conflicts Reveal LLM Prioritiescited by 13
- 2025FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulnesscited by 9
- 2025The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Modelscited by 7
- 2025Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedbackcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Binary Moral Judgment: Modeling Ethical Pluralism in AI
- 2025The Convergent Ethics of AI? Analyzing Moral Foundation Priorities in Large Language Models with a Multi-Framework Approach
- 2026Unsupervised Elicitation of Moral Values from Language Models
- 2024MoralBench: Moral Evaluation of LLMs
- 2025Beyond Ethical Alignment: Evaluating LLMs as Artificial Moral Assistants
- 2025Diverse Human Value Alignment for Large Language Models via Ethical Reasoning
- 2025Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs
- 2026Are LLMs Bad at Moral Reasoning?