Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
AI alignment requires AI systems to adhere to human norms, values, or intentions. Under value pluralism there is no correct target, but a shared prerequisite is that the system's behavior expresses a coherent policy: a mapping from situations to verdicts that is invariant while a situation's morally relevant features are preserved, and sensitive when they change. We introduce four structural conditions for such coherent policies: verdict stability, monotonicity, decisiveness, and Pareto viability. Together they measure a form of moral competence that is evaluable from behavior alone, without reference to a moral standard or expert baseline, forming a structural floor for alignment rather than a normative target. We demonstrate the methodology on three simulated deployments featuring LLM-based agents facing moral dilemmas. Evaluating nine frontier models under a factorial design of five paraphrases, five escalation levels, and three dominance conditions, we show no model expresses a coherent policy across the three deployments: surface-form perturbation alone produces verdict-rate shifts of up to 99 percentage points at a single escalation level, and a model's success on one scenario does not predict its competence on another. This suggests LLM-based agents are not currently the kind of object to which alignment can meaningfully apply.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 40 of its 60 references that are in the map. First posted on arXiv 4 Sep 2026.
Builds on
40 of 60 references.
Closest ancestors · 1
Same direction, newest first
Other references · 39
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
- 2020Aligning AI With Shared Human Valuescited by 723
- 2020Artificial Intelligence, Values and Alignmentcited by 448
- 2018Scalable agent alignment via reward modeling: a research directioncited by 447
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
27 more
- 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
- 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
- 2025Multi-Agent Risks from Advanced AIcited by 163
- 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
- 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
- 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
- 2024Collective Constitutional AI: Aligning a Language Model with Public Inputcited by 123
- 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
- 2024The Ethics of Advanced AI Assistantscited by 107
- 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
- 2024Axioms for AI Alignment from Human Feedbackcited by 52
- 2024Beyond Preferences in AI Alignmentcited by 49
- 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 36
- 2021Hard Choices in Artificial Intelligencecited by 26
- 2018Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)cited by 22
- 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approachcited by 22
- 2025LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Modelscited by 21
- 2026Who's in Charge? Disempowerment Patterns in Real-World LLM Usagecited by 15
- 2025The Value of Disagreement in AI Design, Evaluation, and Alignmentcited by 12
- 2025Distributional AGI Safetycited by 12
- 2025Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMscited by 10
- 2024Representative Social Choice: From Learning Theory to AI Alignmentcited by 8
- 2025Resource Rational Contractualism Should Guide AI Alignmentcited by 8
- 2025Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiencescited by 8
- 2026The Fragility Of Moral Judgment In Large Language Modelscited by 8
- 2026Agentic AI and the next intelligence explosioncited by 6
- 2026RealityTest: How People Probe AI Identity and Whether Models Disclose Itcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Normative Conflicts and Shallow AI Alignment
- 2026MoralityGym: A Benchmark for Evaluating Hierarchical Moral Alignment in Sequential Decision-Making Agents
- 2023Hybrid Approaches for Moral Value Alignment in AI Agents: a Manifesto
- 2024Moral Alignment for LLM Agents
- 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignment
- 2025Beyond Ethical Alignment: Evaluating LLMs as Artificial Moral Assistants
- 2026Are LLMs Bad at Moral Reasoning?
- 2025MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning