# Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity

arXiv: https://arxiv.org/abs/2110.04041 · HTML: https://mlsanity.com/paper/2110.04041/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 15 of 15)

- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Stochastic Neural Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1704.03012/) (2017)
- [The Mechanics of n-Player Differentiable Games](https://mlsanity.com/paper/1802.05642/) (2018)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [Emergent Coordination Through Competition](https://mlsanity.com/paper/1902.07151/) (2019)
- [Social diversity and social preferences in mixed-motive reinforcement learning](https://mlsanity.com/paper/2002.02325/) (2020)
- [Re-evaluating Evaluation](https://mlsanity.com/paper/1806.02643/) (2018)
- [Graphical Models for Game Theory](https://mlsanity.com/paper/1301.2281/) (2013)

## Built on by (most cited first, 22 of 22)

- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [K-level Reasoning for Zero-Shot Coordination in Hanabi](https://mlsanity.com/paper/2207.07166/) (2022)
- [MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.03376/) (2023)
- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [NeuPL: Neural Population Learning](https://mlsanity.com/paper/2202.07415/) (2022)
- [Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS](https://mlsanity.com/paper/2206.05060/) (2022)
- [Lyapunov Exponents for Diversity in Differentiable Games](https://mlsanity.com/paper/2112.14570/) (2021)
- [Simplex Neural Population Learning: Any-Mixture Bayes-Optimality in Symmetric Zero-sum Games](https://mlsanity.com/paper/2205.15879/) (2022)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [Neural Population Learning beyond Symmetric Zero-sum Games](https://mlsanity.com/paper/2401.05133/) (2024)
- [Multi-Agent Diagnostics for Robustness via Illuminated Diversity](https://mlsanity.com/paper/2401.13460/) (2024)
- [Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions](https://mlsanity.com/paper/2411.01166/) (2024)
- [Watts: Infrastructure for Open-Ended Learning](https://mlsanity.com/paper/2204.13250/) (2022)
- [Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmas](https://mlsanity.com/paper/2305.00768/) (2023)
- [GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning](https://mlsanity.com/paper/2601.20753/) (2026)
- [Rethinking Teacher-Student Curriculum Learning through the Cooperative Mechanics of Experience](https://mlsanity.com/paper/2404.03084/) (2024)
- [Robust Agents in Open-Ended Worlds](https://mlsanity.com/paper/2512.08139/) (2025)
- [Principal Trade-off Analysis](https://mlsanity.com/paper/2206.07520/) (2022)
- [Centralized control for multi-agent RL in a complex Real-Time-Strategy game](https://mlsanity.com/paper/2304.13004/) (2023)
- [Adaptive Human-AI Coordination via Hierarchical Action Disentanglement](https://mlsanity.com/paper/2605.24343/) (2026)
- [Partner-Aware Hierarchical Skill Discovery for Robust Human-AI Collaboration](https://mlsanity.com/paper/2605.24352/) (2026)

## Abstract (arXiv)

Strategic diversity is often essential in games: in multi-player games, for example, evaluating a player against a diverse set of strategies will yield a more accurate estimate of its performance. Furthermore, in games with non-transitivities diversity allows a player to cover several winning strategies. However, despite the significance of strategic diversity, training agents that exhibit diverse behaviour remains a challenge. In this paper we study how to construct diverse populations of agents by carefully structuring how individuals within a population interact. Our approach is based on interaction graphs, which control the flow of information between agents during training and can encourage agents to specialise on different strategies, leading to improved overall performance. We provide evidence for the importance of diversity in multi-agent training and analyse the effect of applying different interaction graphs on the training trajectories, diversity and performance of populations in a range of games. This is an extended version of the long abstract published at AAMAS.