# Learning Roles with Emergent Social Value Orientations

arXiv: https://arxiv.org/abs/2301.13812 · HTML: https://mlsanity.com/paper/2301.13812/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 117 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [ROMA: Multi-Agent Reinforcement Learning with Emergent Roles](https://mlsanity.com/paper/2003.08039/) (2020)
- [RODE: Learning Roles to Decompose Multi-Agent Tasks](https://mlsanity.com/paper/2010.01523/) (2020)
- [On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization](https://mlsanity.com/paper/1607.05447/) (2016)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [Learning to Incentivize Other Learning Agents](https://mlsanity.com/paper/2006.06051/) (2020)
- [Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)](https://mlsanity.com/paper/2112.15422/) (2021)
- [Social diversity and social preferences in mixed-motive reinforcement learning](https://mlsanity.com/paper/2002.02325/) (2020)
- [The social dilemma of autonomous vehicles](https://mlsanity.com/paper/1510.03346/) (2015)
- [Prosocial learning agents solve generalized Stag Hunts better than selfish ones](https://mlsanity.com/paper/1709.02865/) (2017)
- [Iteratively-Refined Interactive 3D Medical Image Segmentation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1911.10334/) (2019)
- [Evolving intrinsic motivations for altruistic behavior](https://mlsanity.com/paper/1811.05931/) (2018)
- [Learning Reciprocity in Complex Sequential Social Dilemmas](https://mlsanity.com/paper/1903.08082/) (2019)
- [A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings](https://mlsanity.com/paper/2106.09012/) (2021)
- [Consequentialist conditional cooperation in social dilemmas with imperfect information](https://mlsanity.com/paper/1710.06975/) (2017)
- [Learning Structured Communication for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2002.04235/) (2020)
- [Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferences](https://mlsanity.com/paper/2011.05373/) (2020)
- [Partner Selection for the Emergence of Cooperation in Multi-Agent Systems Using Reinforcement Learning](https://mlsanity.com/paper/1902.03185/) (2019)
- [Adaptive Incentive Design with Multi-Agent Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/2112.10859/) (2021)
- [D3C: Reducing the Price of Anarchy in Multi-Agent Learning](https://mlsanity.com/paper/2010.00575/) (2020)
- [A Generalised Method for Empirical Game Theoretic Analysis](https://mlsanity.com/paper/1803.06376/) (2018)
- [Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences](https://mlsanity.com/paper/2010.09054/) (2020)

## Built on by (most cited first, 2 of 2)

- [Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas](https://mlsanity.com/paper/2402.17270/) (2024)
- [Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.10865/) (2023)

## Abstract (arXiv)

Social dilemmas can be considered situations where individual rationality leads to collective irrationality. The multi-agent reinforcement learning community has leveraged ideas from social science, such as social value orientations (SVO), to solve social dilemmas in complex cooperative tasks. In this paper, by first introducing the typical "division of labor or roles" mechanism in human society, we provide a promising solution for intertemporal social dilemmas (ISD) with SVOs. A novel learning framework, called Learning Roles with Emergent SVOs (RESVO), is proposed to transform the learning of roles into the social value orientation emergence, which is symmetrically solved by endowing agents with altruism to share rewards with other agents. An SVO-based role embedding space is then constructed by individual conditioning policies on roles with a novel rank regularizer and mutual information maximizer. Experiments show that RESVO achieves a stable division of labor and cooperation in ISDs with different complexity.