mlsanityData to 7 Oct 2026

FoX: Formation-aware exploration in multi-agent reinforcement learning

arXiv 2308.11272 ↗First posted 22 Aug 2023cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Recently, deep multi-agent reinforcement learning (MARL) has gained significant popularity due to its success in various cooperative multi-agent tasks. However, exploration still remains a challenging problem in MARL due to the partial observability of the agents and the exploration space that can grow exponentially as the number of agents increases. Firstly, in order to address the scalability issue of the exploration space, we define a formation-based equivalence relation on the exploration space and aim to reduce the search space by exploring only meaningful states in different formations. Then, we propose a novel formation-aware exploration (FoX) framework that encourages partially observable agents to visit the states in diverse formations by guiding them to be well aware of their current formation solely based on their own observations. Numerical results show that the proposed FoX framework significantly outperforms the state-of-the-art MARL algorithms on Google Research Football (GRF) and sparse Starcraft II multi-agent challenge (SMAC) tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

43 of 57 references.

References · 43

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  8. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  12. 2019The StarCraft Multi-Agent Challengecited by 589
31 more
  1. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  2. 2016Deep Exploration via Bootstrapped DQNcited by 518
  3. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  4. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  5. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  6. 2016Learning Multiagent Communication with Backpropagationcited by 355
  7. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  8. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  9. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  10. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  11. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  12. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  13. 2017Count-Based Exploration with Neural Density Modelscited by 255
  14. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  15. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  16. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  17. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  18. 2019Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Controlcited by 159
  19. 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 135
  20. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  21. 2018Count-Based Exploration with the Successor Representationcited by 112
  22. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  23. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 96
  24. 2020Qatten: A General Framework for Cooperative Multiagent Reinforcement Learningcited by 81
  25. 2019Influence-Based Multi-Agent Explorationcited by 75
  26. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  27. 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
  28. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  29. 2019Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learningcited by 33
  30. 2020UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learningcited by 32
  31. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 1.

9 papers.

  1. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
  2. 2025Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learningcited by 5
  3. 2025Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Explorationcited by 3
  4. 2026Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learningcited by 3
  5. 2025Focusing Influence Mechanism for Multi-Agent Reinforcement Learningcited by 2
  6. 2025Enhancing Sample Efficiency in Multi-Agent RL with Uncertainty Quantification and Selective Explorationcited by 1
  7. 2025Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learningcited by 1
  8. 2025AOAD-MAT: Transformer-based multi-agent deep reinforcement learning model considering agents' order of action decisionscited by 1
  9. 2025Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learningcited by 0