# Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2405.15054 · HTML: https://mlsanity.com/paper/2405.15054/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 20 of 20)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [ROMA: Multi-Agent Reinforcement Learning with Emergent Roles](https://mlsanity.com/paper/2003.08039/) (2020)
- [Celebrating Diversity in Shared Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.02195/) (2021)
- [Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing](https://mlsanity.com/paper/2102.07475/) (2021)
- [Influence-Based Multi-Agent Exploration](https://mlsanity.com/paper/1910.05512/) (2019)
- [VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learning](https://mlsanity.com/paper/2207.03530/) (2022)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [Modelling Behavioural Diversity for Learning in Open-Ended Games](https://mlsanity.com/paper/2103.07927/) (2021)
- [Heterogeneous Multi-Robot Reinforcement Learning](https://mlsanity.com/paper/2301.07137/) (2023)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [The Emergence of Individuality](https://mlsanity.com/paper/2006.05842/) (2020)
- [Quantifying the effects of environment and population diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2102.08370/) (2021)
- [System Neural Diversity: Measuring Behavioral Heterogeneity in Multi-Agent Learning](https://mlsanity.com/paper/2305.02128/) (2023)
- [Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL](https://mlsanity.com/paper/2207.05683/) (2022)
- [Mix-ME: Quality-Diversity for Multi-Agent Learning](https://mlsanity.com/paper/2311.01829/) (2023)
- [Fixed Integral Neural Networks](https://mlsanity.com/paper/2307.14439/) (2023)

## Built on by (most cited first, 11 of 11)

- [HyperMARL: Adaptive Hypernetworks for Multi-Agent RL](https://mlsanity.com/paper/2412.04233/) (2024)
- [Predicting Multi-Agent Specialization via Task Parallelizability](https://mlsanity.com/paper/2503.15703/) (2025)
- [Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.15517/) (2024)
- [The impact of behavioral diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2412.16244/) (2024)
- [InvestESG: A multi-agent reinforcement learning benchmark for studying climate investment as a social dilemma](https://mlsanity.com/paper/2411.09856/) (2024)
- [When Is Diversity Rewarded in Cooperative Multi-Agent Learning?](https://mlsanity.com/paper/2506.09434/) (2025)
- [Cooperative Game-Theoretic Credit Assignment for Multi-Agent Policy Gradients via the Core](https://mlsanity.com/paper/2506.04265/) (2025)
- [Events as Triggers for Behavioral Diversity in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.12388/) (2026)
- [Structured Diversity Control: A Dual-Level Framework for Group-Aware Multi-Agent Coordination](https://mlsanity.com/paper/2506.18651/) (2025)
- [Graph-SND: Sparse Aggregation for Behavioral Diversity in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.05020/) (2026)
- [Optimizing Team Behavior via Extremum-seeking Control in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2610.18808/) (2026)

## Abstract (arXiv)

The study of behavioral diversity in Multi-Agent Reinforcement Learning (MARL) is a nascent yet promising field. In this context, the present work deals with the question of how to control the diversity of a multi-agent system. With no existing approaches to control diversity to a set value, current solutions focus on blindly promoting it via intrinsic rewards or additional loss functions, effectively changing the learning objective and lacking a principled measure for it. To address this, we introduce Diversity Control (DiCo), a method able to control diversity to an exact value of a given metric by representing policies as the sum of a parameter-shared component and dynamically scaled per-agent components. By applying constraints directly to the policy architecture, DiCo leaves the learning objective unchanged, enabling its applicability to any actor-critic MARL algorithm. We theoretically prove that DiCo achieves the desired diversity, and we provide several experiments, both in cooperative and competitive tasks, that show how DiCo can be employed as a novel paradigm to increase performance and sample efficiency in MARL. Multimedia results are available on the paper's website: https://sites.google.com/view/dico-marl.