# Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferences

arXiv: https://arxiv.org/abs/2011.05373 · HTML: https://mlsanity.com/paper/2011.05373/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2020.

## Built on by (most cited first, 21 of 21)

- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [Mind the gap: Challenges of deep learning approaches to Theory of Mind](https://mlsanity.com/paper/2203.16540/) (2022)
- [Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration](https://mlsanity.com/paper/2301.03398/) (2023)
- [Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas](https://mlsanity.com/paper/2402.17270/) (2024)
- [Seven Security Challenges in Cross-domain Multi-agent LLM Systems](https://mlsanity.com/paper/2505.23847/) (2025)
- [The Importance of Credo in Multiagent Learning](https://mlsanity.com/paper/2204.07471/) (2022)
- [A multi-agent reinforcement learning model of reputation and cooperation in human groups](https://mlsanity.com/paper/2103.04982/) (2021)
- [Meta-Value Learning: a General Framework for Learning with Learning Awareness](https://mlsanity.com/paper/2307.08863/) (2023)
- [Exploring the Benefits of Teams in Multiagent Learning](https://mlsanity.com/paper/2205.02328/) (2022)
- [Mediated Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.08419/) (2023)
- [Normative Modules: A Generative Agent Architecture for Learning Norms that Supports Multi-Agent Cooperation](https://mlsanity.com/paper/2405.19328/) (2024)
- [Best Response Shaping](https://mlsanity.com/paper/2404.06519/) (2024)
- [Learning Roles with Emergent Social Value Orientations](https://mlsanity.com/paper/2301.13812/) (2023)
- [Multi-AI Complex Systems in Humanitarian Response](https://mlsanity.com/paper/2208.11282/) (2022)
- [Intrinsic fluctuations of reinforcement learning promote cooperation](https://mlsanity.com/paper/2209.01013/) (2022)
- [Dynamics of Moral Behavior in Heterogeneous Populations of Learning Agents](https://mlsanity.com/paper/2403.04202/) (2024)
- [PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Training](https://mlsanity.com/paper/2509.06053/) (2025)
- [CooT: Learning to Coordinate In-Context with Coordination Transformers](https://mlsanity.com/paper/2506.23549/) (2025)
- [Tapes Together Strong: The Co-evolution of Computation and Cooperation](https://mlsanity.com/paper/2609.10817/) (2026)

## Abstract (arXiv)

Multi-agent reinforcement learning (MARL) has shown recent success in increasingly complex fixed-team zero-sum environments. However, the real world is not zero-sum nor does it have fixed teams; humans face numerous social dilemmas and must learn when to cooperate and when to compete. To successfully deploy agents into the human world, it may be important that they be able to understand and help in our conflicts. Unfortunately, selfish MARL agents typically fail when faced with social dilemmas. In this work, we show evidence of emergent direct reciprocity, indirect reciprocity and reputation, and team formation when training agents with randomized uncertain social preferences (RUSP), a novel environment augmentation that expands the distribution of environments agents play in. RUSP is generic and scalable; it can be applied to any multi-agent environment without changing the original underlying game dynamics or objectives. In particular, we show that with RUSP these behaviors can emerge and lead to higher social welfare equilibria in both classic abstract social dilemmas like Iterated Prisoner's Dilemma as well in more complex intertemporal environments.