# Optimizing Team Behavior via Extremum-seeking Control in Multi-agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2610.18808 · HTML: https://mlsanity.com/paper/2610.18808/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 26 of 26)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [ROMA: Multi-Agent Reinforcement Learning with Emergent Roles](https://mlsanity.com/paper/2003.08039/) (2020)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.07169/) (2020)
- [Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing](https://mlsanity.com/paper/2102.07475/) (2021)
- [VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learning](https://mlsanity.com/paper/2207.03530/) (2022)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [BenchMARL: Benchmarking Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2312.01472/) (2023)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2303.00912/) (2023)
- [System Neural Diversity: Measuring Behavioral Heterogeneity in Multi-Agent Learning](https://mlsanity.com/paper/2305.02128/) (2023)
- [Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.15054/) (2024)
- [HyperMARL: Adaptive Hypernetworks for Multi-Agent RL](https://mlsanity.com/paper/2412.04233/) (2024)
- [Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2410.08540/) (2024)
- [Reward-Sharing Relational Networks in Multi-Agent Reinforcement Learning as a Framework for Emergent Behavior](https://mlsanity.com/paper/2207.05886/) (2022)
- [Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning](https://mlsanity.com/paper/2404.08233/) (2024)
- [Model-free online motion adaptation for energy efficient flights of multicopters](https://mlsanity.com/paper/2108.03807/) (2021)
- [When Is Diversity Rewarded in Cooperative Multi-Agent Learning?](https://mlsanity.com/paper/2506.09434/) (2025)
- [Influence of Team Interactions on Multi-Robot Cooperation: A Relational Network Perspective](https://mlsanity.com/paper/2310.12910/) (2023)
- [Extremum-Seeking Action Selection for Accelerating Policy Optimization](https://mlsanity.com/paper/2404.01598/) (2024)
- [Events as Triggers for Behavioral Diversity in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.12388/) (2026)

## Abstract (arXiv)

In cooperative Multi-Agent Reinforcement Learning (MARL), agents must learn to coordinate their actions to achieve shared objectives. A key indicator of effective coordination is behavioral diversity, i.e., the agents' potential to enact distinct yet complementary control policies given the same set of observations. Although behavioral diversity can enable role specialization and richer joint behaviors, its appropriate level is highly context- and task-dependent and often costly to tune. We propose Adaptive Diversity Control (ADiCo), a two-timescale framework that adapts behavioral diversity online during training. On a fast timescale, Diversity Control (DiCo) regulates system diversity toward a target value. On a slower timescale, an Extremum Seeking Controller (ESC) adjusts this target using only mean episodic returns, without requiring an explicit model or analytical gradient of the diversity-performance landscape. We evaluate ADiCo on six configurations across two cooperative MARL tasks, where it maintains task-appropriate diversity levels: low diversity when homogeneous policies are beneficial and higher diversity when heterogeneous policies are required. Through this adaptation, ADiCo matches or exceeds the mean performance of DiCo (best grid-searched diversity target) and Unconstrained DiCo (diversity allowed to emerge without target regulation). These results demonstrate that incorporating classical feedback control into MARL training enables effective coordination by providing a systematic alternative to task-specific manual selection of the diversity target.