mlsanityData to 14 Oct 2026

Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learning

arXiv 2502.03506 ↗First posted 5 Feb 2025cs.MA cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The Centralized Training with Decentralized Execution (CTDE) paradigm is widely used in cooperative multi-agent reinforcement learning. However, conventional methods based on CTDE can suffer from value underestimation and converge to suboptimal solutions. While such underestimation is typically attributed to the representational limitations of monotonic structures, we provide a novel perspective by demonstrating that the insufficient sampling of optimal joint actions during exploration is also a critical factor. To address this problem, we propose Optimistic ε-Greedy Exploration. Our method introduces optimistic action-value networks that serve as decoupled exploration indicators, which we theoretically prove to converge in probability to the maximum achievable returns. By sampling actions from these distributions with a probability of ε, we effectively increase the selection frequency of high-return joint actions. Experimental results in various environments reveal that our strategy effectively prevents the algorithm from falling into suboptimal solutions and significantly improves final returns, win rates, and convergence speeds compared to other enhanced algorithms. Our code has been open-sourced at https://github.com/qxqxtxdy/OptimisticExploration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

27 of 37 references.

References · 27

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  2. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  3. 2019The StarCraft Multi-Agent Challengecited by 597
  4. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  5. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  6. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  7. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  8. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 261
  9. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  10. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  11. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 194
  12. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 117
15 more
  1. 2019Influence-Based Multi-Agent Explorationcited by 75
  2. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  3. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
  4. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  5. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 50
  6. 2022CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender Systemcited by 38
  7. 2021DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learningcited by 26
  8. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  9. 2022Pareto Actor-Critic for Equilibrium Selection in Multi-Agent Reinforcement Learningcited by 14
  10. 2019Exploration with Unreliable Intrinsic Reward in Multi-Agent Reinforcement Learningcited by 13
  11. 2020A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learningcited by 11
  12. 2023Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learningcited by 5
  13. 2022Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learningcited by 3
  14. 2022Sampling Efficient Deep Reinforcement Learning through Preference-Guided Stochastic Explorationcited by 2
  15. 2019ColosseumRL: A Framework for Multiagent Reinforcement Learning in N-Player Gamescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimism as Risk-Seeking in Multi-Agent Reinforcement Learning
  2. 2024Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling
  3. 2026Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning
  4. 2024Beyond Optimism: Exploration With Partially Observable Rewards
  5. 2025Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
  6. 2026The Horizon Threshold in Cooperative Multi-Agent Reward-Free Exploration
  7. 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies
  8. 2024MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structure