# Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2301.08491 · HTML: https://mlsanity.com/paper/2301.08491/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 75 references that are in the map.
- First posted on arXiv 20 Jan 2023.

## Builds on (most cited first, 14 of 14)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [Social diversity and social preferences in mixed-motive reinforcement learning](https://mlsanity.com/paper/2002.02325/) (2020)
- [Prosocial learning agents solve generalized Stag Hunts better than selfish ones](https://mlsanity.com/paper/1709.02865/) (2017)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Consequentialist conditional cooperation in social dilemmas with imperfect information](https://mlsanity.com/paper/1710.06975/) (2017)
- [Partner Selection for the Emergence of Cooperation in Multi-Agent Systems Using Reinforcement Learning](https://mlsanity.com/paper/1902.03185/) (2019)
- [Building Ethics into Artificial Intelligence](https://mlsanity.com/paper/1812.02953/) (2018)
- [Responsible Autonomy](https://mlsanity.com/paper/1706.02513/) (2017)
- [Cooperation and Reputation Dynamics with Reinforcement Learning](https://mlsanity.com/paper/2102.07523/) (2021)
- [Reinforcement Learning Produces Dominant Strategies for the Iterated Prisoner's Dilemma](https://mlsanity.com/paper/1707.06307/) (2017)

## Built on by (most cited first, 10 of 10)

- [Moral Alignment for LLM Agents](https://mlsanity.com/paper/2410.01639/) (2024)
- [The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process](https://mlsanity.com/paper/2506.01080/) (2025)
- [When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas](https://mlsanity.com/paper/2505.19212/) (2025)
- [Hybrid Approaches for Moral Value Alignment in AI Agents: a Manifesto](https://mlsanity.com/paper/2312.01818/) (2023)
- [Reinforcement Learning and Machine ethics:a systematic review](https://mlsanity.com/paper/2407.02425/) (2024)
- [Steps Towards Satisficing Distributed Dynamic Team Trust](https://mlsanity.com/paper/2309.05378/) (2023)
- [Dynamics of Moral Behavior in Heterogeneous Populations of Learning Agents](https://mlsanity.com/paper/2403.04202/) (2024)
- [Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.02654/) (2024)
- [Building Altruistic and Moral AI Agent with Brain-inspired Emotional Empathy Mechanisms](https://mlsanity.com/paper/2410.21882/) (2024)
- [Achieving Collective Welfare in Multi-Agent Reinforcement Learning via Suggestion Sharing](https://mlsanity.com/paper/2412.12326/) (2024)

## Abstract (arXiv)

Practical uses of Artificial Intelligence (AI) in the real world have demonstrated the importance of embedding moral choices into intelligent agents. They have also highlighted that defining top-down ethical constraints on AI according to any one type of morality is extremely challenging and can pose risks. A bottom-up learning approach may be more appropriate for studying and developing ethical behavior in AI agents. In particular, we believe that an interesting and insightful starting point is the analysis of emergent behavior of Reinforcement Learning (RL) agents that act according to a predefined set of moral rewards in social dilemmas. In this work, we present a systematic analysis of the choices made by intrinsically-motivated RL agents whose rewards are based on moral theories. We aim to design reward structures that are simplified yet representative of a set of key ethical systems. Therefore, we first define moral reward functions that distinguish between consequence- and norm-based agents, between morality based on societal norms or internal virtues, and between single- and mixed-virtue (e.g., multi-objective) methodologies. Then, we evaluate our approach by modeling repeated dyadic interactions between learning moral agents in three iterated social dilemma games (Prisoner's Dilemma, Volunteer's Dilemma and Stag Hunt). We analyze the impact of different types of morality on the emergence of cooperation, defection or exploitation, and the corresponding social outcomes. Finally, we discuss the implications of these findings for the development of moral agents in artificial and mixed human-AI societies.