mlsanityData to 14 Oct 2026

Heterogeneous-Agent Reinforcement Learning

arXiv 2304.09870 ↗First posted 19 Apr 2023cs.LG cs.AI cs.MAcited by 69

Not on the map. Why?

Abstract (arXiv)

The necessity for cooperation among intelligent machines has popularised cooperative multi-agent reinforcement learning (MARL) in AI research. However, many research endeavours heavily rely on parameter sharing among agents, which confines them to only homogeneous-agent setting and leads to training instability and lack of convergence guarantees. To achieve effective cooperation in the general heterogeneous-agent setting, we propose Heterogeneous-Agent Reinforcement Learning (HARL) algorithms that resolve the aforementioned issues. Central to our findings are the multi-agent advantage decomposition lemma and the sequential update scheme. Based on these, we develop the provably correct Heterogeneous-Agent Trust Region Learning (HATRL), and derive HATRPO and HAPPO by tractable approximations. Furthermore, we discover a novel framework named Heterogeneous-Agent Mirror Learning (HAML), which strengthens theoretical guarantees for HATRPO and HAPPO and provides a general template for cooperative MARL algorithmic designs. We prove that all algorithms derived from HAML inherently enjoy monotonic improvement of joint return and convergence to Nash Equilibrium. As its natural outcome, HAML validates more novel algorithms in addition to HATRPO and HAPPO, including HAA2C, HADDPG, and HATD3, which generally outperform their existing MA-counterparts. We comprehensively test HARL algorithms on six challenging benchmarks and demonstrate their superior effectiveness and stability for coordinating heterogeneous agents compared to strong baselines such as MAPPO and QMIX.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Builds on 44 of its 64 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

44 of 64 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  9. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  10. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  11. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  12. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
32 more
  1. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  2. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  3. 2019The StarCraft Multi-Agent Challengecited by 597
  4. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  5. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  6. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 294
  7. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 263
  8. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  9. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 231
  10. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 220
  11. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  12. 2018Mean Field Multi-Agent Reinforcement Learningcited by 207
  13. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  14. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  15. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 122
  16. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  17. 2020Game-Theoretic Multiagent Reinforcement Learningcited by 94
  18. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  19. 2021Settling the Variance of Multi-Agent Policy Gradientscited by 45
  20. 2017Guided Deep Reinforcement Learning for Swarm Systemscited by 44
  21. 2019Bi-level Actor-Critic for Multi-agent Coordinationcited by 39
  22. 2020EigenGame: PCA as a Nash Equilibriumcited by 39
  23. 2020Multi-Agent Determinantal Q-Learningcited by 36
  24. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  25. 2021Learning in Nonzero-Sum Stochastic Games with Potentialscited by 26
  26. 2019Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoningcited by 24
  27. 2021MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learningcited by 23
  28. 2020Multi-Agent Trust Region Policy Optimizationcited by 20
  29. 2021Coordinated Proximal Policy Optimizationcited by 16
  30. 2019Multiagent Rollout Algorithms and Reinforcement Learningcited by 10
  31. 2021WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPUcited by 10
  32. 2021A Game-Theoretic Approach to Multi-Agent Trust Region Optimizationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 28; 2026: 30.

69 papers.

Sort:

By area:

  1. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  2. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 15
  3. 2024Hierarchical Consensus-Based Multi-Agent Reinforcement Learning for Multi-Robot Cooperation Taskscited by 13
  4. 2024HyperMARL: Adaptive Hypernetworks for Multi-Agent RLcited by 11
  5. 2024Sable: a Performant, Efficient and Scalable Sequence Model for MARLcited by 10
  6. 2024Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluationcited by 9
  7. 2025Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspectivecited by 9
  8. 2025HiMA-Ecom: Enabling Joint Training of Hierarchical Multi-Agent E-commerce Assistantscited by 9
  9. 2024SustainDC: Benchmarking for Sustainable Data Center Controlcited by 8
  10. 2025OM2P: Offline Multi-Agent Mean-Flow Policycited by 8
  11. 2024Aligning Individual and Collective Objectives in Multi-Agent Cooperationcited by 7
  12. 2024Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibriumcited by 6
38 more
  1. 2024Coordination Failure in Cooperative Offline MARLcited by 5
  2. 2025SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Controlcited by 4
  3. 2025Joint Computation Offloading and Resource Allocation for Uncertain Maritime MEC via Cooperation of UAVs and Vesselscited by 3
  4. 2025Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learningcited by 3
  5. 2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policiescited by 3
  6. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environmentscited by 2
  7. 2026Guided Collaboration in Heterogeneous LLM-Based Multi-Agent Systems via Entropy-Based Understanding Assessment and Experience Retrievalcited by 2
  8. 2026Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsamplingcited by 2
  9. 2026Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensuscited by 2
  10. 2024Reinforcement Learning Driven Multi-Robot Exploration via Explicit Communication and Density-Based Frontier Searchcited by 1
  11. 2025Generative AI-Enhanced Cooperative MEC of UAVs and Ground Stations for Unmanned Surface Vehiclescited by 1
  12. 2025Learning Dexterous Bimanual Catch Skills through Adversarial-Cooperative Heterogeneous-Agent Reinforcement Learningcited by 1
  13. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 1
  14. 2025The Heterogeneous Multi-Agent Challengecited by 1
  15. 2025HARL-A: An Extensible Benchmark Framework for Heterogeneous Multi-Agent Adversarial Reinforcement Learning in IsaacLabcited by 1
  16. 2025LLMs as Policy-Agnostic Teammates: A Case Study in Human Proxy Design for Heterogeneous Agent Teamscited by 1
  17. 2025Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learningcited by 1
  18. 2025From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demoscited by 1
  19. 2025Heterogeneous Multi-Agent Proximal Policy Optimization for Power Distribution System Restorationcited by 1
  20. 2025Hierarchical Task Offloading and Trajectory Optimization in Low-Altitude Intelligent Networks Via Auction and Diffusion-based MARLcited by 1
  21. 2026Safe Heterogeneous Multi-Agent RL with Communication Regularization for Coordinated Target Acquisitioncited by 1
  22. 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learningcited by 1
  23. 2026RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Trainingcited by 1
  24. 2024Computing Ex Ante Equilibrium in Heterogeneous Zero-Sum Team Gamescited by 0
  25. 2025Policy Optimization and Multi-agent Reinforcement Learning for Mean-variance Team Stochastic Gamescited by 0
  26. 2025LOD: Latent Objective Discovery in Heterogeneous Multi-Agent Reinforcement Learningcited by 0
  27. 2025Embedded Mean Field Reinforcement Learning for Perimeter-defense Gamecited by 0
  28. 2025Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARLcited by 0
  29. 2025Multi-Agent Reinforcement Learning for Autonomous Multi-Satellite Earth Observation: A Realistic Case Studycited by 0
  30. 2025Remembering the Markov Property in Cooperative MARLcited by 0
  31. 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policiescited by 0
  32. 2025Local-Canonicalization Equivariant Graph Neural Networks for Sample-Efficient and Generalizable Swarm Robot Controlcited by 0
  33. 2025Fully Decentralized Cooperative Multi-Agent Reinforcement Learning is A Context Modeling Problemcited by 0
  34. 2025An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithmscited by 0
  35. 2025Independent policy gradient-based reinforcement learning for economic and reliable energy management of multi-microgrid systemscited by 0
  36. 2026Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learningcited by 0
  37. 2026Hierarchical Lead Critic based Multi-Agent Reinforcement Learningcited by 0
  38. 2026HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimizationcited by 0

50 most cited of 69 All 69 (JSON)