# HyperMARL: Adaptive Hypernetworks for Multi-Agent RL

arXiv: https://arxiv.org/abs/2412.04233 · HTML: https://mlsanity.com/paper/2412.04233/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 30 of 37)

- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020)
- [ROMA: Multi-Agent Reinforcement Learning with Emergent Roles](https://mlsanity.com/paper/2003.08039/) (2020)
- [RODE: Learning Roles to Decompose Multi-Agent Tasks](https://mlsanity.com/paper/2010.01523/) (2020)
- [Learning the Pareto Front with Hypernetworks](https://mlsanity.com/paper/2010.04104/) (2020)
- [Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.07169/) (2020)
- [Celebrating Diversity in Shared Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.02195/) (2021)
- [Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing](https://mlsanity.com/paper/2102.07475/) (2021)
- [Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.09870/) (2023)
- [VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learning](https://mlsanity.com/paper/2207.03530/) (2022)
- [JaxMARL: Multi-Agent RL Environments and Algorithms in JAX](https://mlsanity.com/paper/2311.10090/) (2023)
- [Empirical Design in Reinforcement Learning](https://mlsanity.com/paper/2304.01315/) (2023)
- [Settling the Variance of Multi-Agent Policy Gradients](https://mlsanity.com/paper/2108.08612/) (2021)
- [Continual Model-Based Reinforcement Learning with Hypernetworks](https://mlsanity.com/paper/2009.11997/) (2020)
- [Hypernetworks in Meta-Reinforcement Learning](https://mlsanity.com/paper/2210.11348/) (2022)
- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [The Emergence of Individuality](https://mlsanity.com/paper/2006.05842/) (2020)
- [Quantifying the effects of environment and population diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2102.08370/) (2021)
- [Principled Weight Initialization for Hypernetworks](https://mlsanity.com/paper/2312.08399/) (2023)
- [Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity](https://mlsanity.com/paper/2207.14138/) (2022)
- [Recurrent Hypernetworks are Surprisingly Strong in Meta-RL](https://mlsanity.com/paper/2309.14970/) (2023)

## Built on by (most cited first, 11 of 11)

- [Capability-Aware Shared Hypernetworks for Flexible Heterogeneous Multi-Robot Coordination](https://mlsanity.com/paper/2501.06058/) (2025)
- [When Is Diversity Rewarded in Cooperative Multi-Agent Learning?](https://mlsanity.com/paper/2506.09434/) (2025)
- [Probing Dec-POMDP Reasoning in Cooperative MARL](https://mlsanity.com/paper/2602.20804/) (2026)
- [Benchmarking Open-Ended Multi-Agent Coordination in Language Agents](https://mlsanity.com/paper/2606.08340/) (2026)
- [Prism: Spectral Parameter Sharing for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.06476/) (2026)
- [Events as Triggers for Behavioral Diversity in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.12388/) (2026)
- [Wireless Communication Enhanced Value Decomposition for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2604.08728/) (2026)
- [Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.01865/) (2026)
- [Graph-SND: Sparse Aggregation for Behavioral Diversity in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.05020/) (2026)
- [Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning](https://mlsanity.com/paper/2607.09422/) (2026)
- [Optimizing Team Behavior via Extremum-seeking Control in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2610.18808/) (2026)

## Abstract (arXiv)

Adaptive cooperation in multi-agent reinforcement learning (MARL) requires policies to express homogeneous, specialised, or mixed behaviours, yet achieving this adaptivity remains a critical challenge. While parameter sharing (PS) is standard for efficient learning, it notoriously suppresses the behavioural diversity required for specialisation. This failure is largely due to cross-agent gradient interference, a problem we find is surprisingly exacerbated by the common practice of coupling agent IDs with observations. Existing remedies typically add complexity through altered objectives, manual preset diversity levels, or sequential updates -- raising a fundamental question: can shared policies adapt without these intricacies? We propose a solution built on a key insight: an agent-conditioned hypernetwork can generate agent-specific parameters and decouple observation- and agent-conditioned gradients, directly countering the interference from coupling agent IDs with observations. Our resulting method, HyperMARL, avoids the complexities of prior work and empirically reduces policy gradient variance. Across diverse MARL benchmarks (22 scenarios, up to 30 agents), HyperMARL achieves performance competitive with six key baselines while preserving behavioural diversity comparable to non-parameter sharing methods, establishing it as a versatile and principled approach for adaptive MARL. The code is publicly available at https://github.com/KaleabTessera/HyperMARL.