# Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmas

arXiv: https://arxiv.org/abs/2305.00768 · HTML: https://mlsanity.com/paper/2305.00768/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 1 May 2023.

## Builds on (most cited first, 24 of 24)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Multi-task Deep Reinforcement Learning with PopArt](https://mlsanity.com/paper/1809.04474/) (2018)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [Off-Belief Learning](https://mlsanity.com/paper/2103.04000/) (2021)
- [Social diversity and social preferences in mixed-motive reinforcement learning](https://mlsanity.com/paper/2002.02325/) (2020)
- [Maintaining cooperation in complex social dilemmas using deep reinforcement learning](https://mlsanity.com/paper/1707.01068/) (2017)
- [Melting Pot 2.0](https://mlsanity.com/paper/2211.13746/) (2022)
- [Modelling Behavioural Diversity for Learning in Open-Ended Games](https://mlsanity.com/paper/2103.07927/) (2021)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)
- [Evolving intrinsic motivations for altruistic behavior](https://mlsanity.com/paper/1811.05931/) (2018)
- [Consequentialist conditional cooperation in social dilemmas with imperfect information](https://mlsanity.com/paper/1710.06975/) (2017)
- [Quantifying the effects of environment and population diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2102.08370/) (2021)
- [Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity](https://mlsanity.com/paper/2110.04041/) (2021)
- [Adaptable Agent Populations via a Generative Model of Policies](https://mlsanity.com/paper/2107.07506/) (2021)
- [Warmth and competence in human-agent cooperation](https://mlsanity.com/paper/2201.13448/) (2022)
- [Statistical discrimination in learning agents](https://mlsanity.com/paper/2110.11404/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas](https://mlsanity.com/paper/2402.17270/) (2024)

## Abstract (arXiv)

In social psychology, Social Value Orientation (SVO) describes an individual's propensity to allocate resources between themself and others. In reinforcement learning, SVO has been instantiated as an intrinsic motivation that remaps an agent's rewards based on particular target distributions of group reward. Prior studies show that groups of agents endowed with heterogeneous SVO learn diverse policies in settings that resemble the incentive structure of Prisoner's dilemma. Our work extends this body of results and demonstrates that (1) heterogeneous SVO leads to meaningfully diverse policies across a range of incentive structures in sequential social dilemmas, as measured by task-specific diversity metrics; and (2) learning a best response to such policy diversity leads to better zero-shot generalization in some situations. We show that these best-response agents learn policies that are conditioned on their co-players, which we posit is the reason for improved zero-shot generalization results.