mlsanityData to 14 Oct 2026

When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas

arXiv 2505.19212 ↗First posted 25 May 2025cs.CL cs.AI cs.CYcited by 15

Not on the map. Why?

Abstract (arXiv)

Recent advances in LLMs have enabled their use in complex agentic roles, involving decision-making with humans or other agents, making ethical alignment a critical concern. While prior work has examined LLMs' moral judgment and strategic behavior separately, there is limited understanding of how they act when moral imperatives directly conflict with profit incentives. We introduce msimfull (msim) to evaluate how LLMs behave in the prisoner's dilemma and public goods game embedded in morally charged contexts, varying moral framing, opponent behavior, and survival pressure across nine models. Beyond measuring behavior, we estimate the causal effect of each factor via average treatment effects (ATEs) and analyze agents' own reasoning traces to characterize the motives behind their choices. We find that no model remains consistently moral, with cooperation rates ranging from 7.9% to 76.3%. Game structure and moral framing are the strongest causal drivers of moral behavior, while reasoning-trace analysis reveals distinct motive profiles across models, ranging from predominantly payoff-maximizing to moral- and reputation-oriented. Together, these results expose the situational brittleness of current LLMs' moral behavior and the risk of deploying them where profit incentives conflict with ethical guidelines.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

33 of 59 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  11. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  12. 2024Alignment faking in large language modelscited by 360
21 more
  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  2. 2023Playing repeated games with Large Language Modelscited by 217
  3. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  4. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  5. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  6. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
  7. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 125
  8. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 113
  9. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  10. 2023Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysiscited by 92
  11. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 86
  12. 2023Strategic Reasoning with Language Modelscited by 64
  13. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 58
  14. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 56
  15. 2024MoralBench: Moral Evaluation of LLMscited by 50
  16. 2024Moral Alignment for LLM Agentscited by 39
  17. 2023Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?cited by 36
  18. 2025Will Systems of LLM Agents Cooperate: An Investigation into a Social Dilemmacited by 25
  19. 2024Paraphrase Types Elicit Prompt Engineering Capabilitiescited by 15
  20. 2023Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learningcited by 10
  21. 2025Spontaneous Giving and Calculated Greed in Language Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamicscited by 19
  2. 2026CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmascited by 16
  3. 2025The Role of Social Learning and Collective Norm Formation in Fostering Cooperation in LLM Multi-Agent Systemscited by 9
  4. 2026Evaluating Collective Behaviour of Hundreds of LLM Agentscited by 8
  5. 2025Survival at Any Cost? LLMs and the Choice Between Self-Preservation and Human Harmcited by 3
  6. 2025Social Welfare Function Leaderboard: When LLM Agents Allocate Social Welfarecited by 2
  7. 2025The AI in the Mirror: LLM Self-Recognition in an Iterated Public Goods Gamecited by 1
  8. 2026Payoff scaling shapes cooperation in LLM agents across languagescited by 1
  9. 2026To Nuke or Not to Nuke: LLMs' (Missing) Ethical Reasoning and Actions in a High-Stakes Decision-Making Simulationcited by 1
  10. 2026Predicting Decisions of AI Agents from Limited Interaction through Text-Tabular Modelingcited by 0
  11. 2026Is Lying an Emergent Behaviour in LLMs? Evidence from Gaslighting AI agents in a Sustainability Gamecited by 0
  12. 2026When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Gamescited by 0
3 more
  1. 2026Causal Behavioral Evaluation of AI Agents at Scale via Automated Behavioral Sciencecited by 0
  2. 2026From Certain Doom to Survival: Agent-Driven Self-Governance in LLM Agent Societiescited by 0
  3. 2026Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Many LLMs Are More Utilitarian Than One
  2. 2025Corrupted by Reasoning: Reasoning Language Models Become Free-Riders in Public Goods Games
  3. 2026Social Catalysts, Not Moral Agents: The Illusion of Alignment in LLM Societies
  4. 2025The Staircase of Ethics: Probing LLM Value Priorities through Multi-Step Induction to Complex Moral Dilemmas
  5. 2025Understanding LLM Agent Behaviours via Game Theory: Strategy Recognition, Biases and Multi-Agent Dynamics
  6. 2025"Pull or Not to Pull?'': Investigating Moral Biases in Leading Large Language Models Across Ethical Dilemmas
  7. 2025The Morality of Probability: How Implicit Moral Biases in LLMs May Shape the Future of Human-AI Symbiosis
  8. 2026Do LLMs Take Care of Their Own? Similarity Signals Can Induce Cooperation