mlsanityData to 14 Oct 2026

MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structure

arXiv 2405.00902 ↗First posted 1 May 2024cs.LG cs.AI cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-agent reinforcement learning (MARL) algorithms often struggle to find strategies close to Pareto optimal Nash Equilibrium, owing largely to the lack of efficient exploration. The problem is exacerbated in sparse-reward settings, caused by the larger variance exhibited in policy learning. This paper introduces MESA, a novel meta-exploration method for cooperative multi-agent learning. It learns to explore by first identifying the agents' high-rewarding joint state-action subspace from training tasks and then learning a set of diverse exploration policies to "cover" the subspace. These trained exploration policies can be integrated with any off-policy MARL algorithm for test-time tasks. We first showcase MESA's advantage in a multi-step matrix game. Furthermore, experiments show that with learned exploration policies, MESA achieves significantly better performance in sparse-reward tasks in several multi-agent particle environments and multi-agent MuJoCo environments, and exhibits the ability to generalize to more challenging tasks at test time.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

38 of 45 references.

References · 38

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2018Exploration by Random Network Distillationcited by 993
  6. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  10. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  11. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  12. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
26 more
  1. 2020First return, then explorecited by 266
  2. 2017Count-Based Exploration with Neural Density Modelscited by 257
  3. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  4. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
  5. 2019MAVEN: Multi-Agent Variational Explorationcited by 185
  6. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 178
  7. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  8. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  9. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  10. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
  11. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  12. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 130
  13. 2018Inequity aversion improves cooperation in intertemporal social dilemmascited by 106
  14. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  15. 2019Influence-Based Multi-Agent Explorationcited by 75
  16. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  17. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 53
  18. 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
  19. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  20. 2020UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learningcited by 33
  21. 2020MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Explorationcited by 25
  22. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  23. 2019Meta Reinforcement Learning with Task Embedding and Shared Policycited by 15
  24. 2018Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learningcited by 13
  25. 2018Learning to Explore with Meta-Policy Gradientcited by 9
  26. 2019Quadratic Q-network for Learning Continuous Control for Autonomous Vehiclescited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MIR: Efficient Exploration in Episodic Multi-Agent Reinforcement Learning via Mutual Intrinsic Reward
  2. 2024MARL-LNS: Cooperative Multi-agent Reinforcement Learning via Large Neighborhoods Search
  3. 2025Multi-Agent Guided Policy Search for Non-Cooperative Dynamic Games
  4. 2025An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks
  5. 2024Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey
  6. 2025Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning
  7. 2025Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
  8. 2025Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVs