# Cooperation and Reputation Dynamics with Reinforcement Learning

arXiv: https://arxiv.org/abs/2102.07523 · HTML: https://mlsanity.com/paper/2102.07523/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 40 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 6 of 6)

- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [Stable Opponent Shaping in Differentiable Games](https://mlsanity.com/paper/1811.08469/) (2018)
- [Prosocial learning agents solve generalized Stag Hunts better than selfish ones](https://mlsanity.com/paper/1709.02865/) (2017)
- [Learning Reciprocity in Complex Sequential Social Dilemmas](https://mlsanity.com/paper/1903.08082/) (2019)
- [Partner Selection for the Emergence of Cooperation in Multi-Agent Systems Using Reinforcement Learning](https://mlsanity.com/paper/1902.03185/) (2019)

## Built on by (most cited first, 13 of 13)

- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2301.08491/) (2023)
- [Reputation as a Solution to Cooperation Collapse in LLM-based MASs](https://mlsanity.com/paper/2505.05029/) (2025)
- [The Importance of Credo in Multiagent Learning](https://mlsanity.com/paper/2204.07471/) (2022)
- [Aligning Individual and Collective Objectives in Multi-Agent Cooperation](https://mlsanity.com/paper/2402.12416/) (2024)
- [Exploring the Benefits of Teams in Multiagent Learning](https://mlsanity.com/paper/2205.02328/) (2022)
- [Investigating the Impact of Direct Punishment on the Emergence of Cooperation in Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2301.08278/) (2023)
- [Emergent Cooperation under Uncertain Incentive Alignment](https://mlsanity.com/paper/2401.12646/) (2024)
- [Learning Roles with Emergent Social Value Orientations](https://mlsanity.com/paper/2301.13812/) (2023)
- [Steps Towards Satisficing Distributed Dynamic Team Trust](https://mlsanity.com/paper/2309.05378/) (2023)
- [State-of-the-art in Robot Learning for Multi-Robot Collaboration: A Comprehensive Survey](https://mlsanity.com/paper/2408.11822/) (2024)
- [The Reciprocity Gradient](https://mlsanity.com/paper/2605.08323/) (2026)
- [The Role of Network Topology and Opponent Information in Shaping Cooperation in Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2608.28977/) (2026)

## Abstract (arXiv)

Creating incentives for cooperation is a challenge in natural and artificial systems. One potential answer is reputation, whereby agents trade the immediate cost of cooperation for the future benefits of having a good reputation. Game theoretical models have shown that specific social norms can make cooperation stable, but how agents can independently learn to establish effective reputation mechanisms on their own is less understood. We use a simple model of reinforcement learning to show that reputation mechanisms generate two coordination problems: agents need to learn how to coordinate on the meaning of existing reputations and collectively agree on a social norm to assign reputations to others based on their behavior. These coordination problems exhibit multiple equilibria, some of which effectively establish cooperation. When we train agents with a standard Q-learning algorithm in an environment with the presence of reputation mechanisms, convergence to undesirable equilibria is widespread. We propose two mechanisms to alleviate this: (i) seeding a proportion of the system with fixed agents that steer others towards good equilibria; and (ii), intrinsic rewards based on the idea of introspection, i.e., augmenting agents' rewards by an amount proportionate to the performance of their own strategy against themselves. A combination of these simple mechanisms is successful in stabilizing cooperation, even in a fully decentralized version of the problem where agents learn to use and assign reputations simultaneously. We show how our results relate to the literature in Evolutionary Game Theory, and discuss implications for artificial, human and hybrid systems, where reputations can be used as a way to establish trust and cooperation.