Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies
Not on the map. Why?
Abstract (arXiv)
Independent on-policy policy gradient algorithms are widely used for multi-agent reinforcement learning (MARL) in cooperative and no-conflict games, but they are known to converge sub-optimally when each agent's individual policy gradient points away from an optimal joint equilibrium. Going beyond prior work, we observe that sub-optimal convergence can still arise even when the expected individual policy gradients of each agent point toward the optimal joint solution. After collecting a finite set of trajectories, stochasticity in independent action sampling can cause the joint data distribution to deviate from the expected joint on-policy distribution. This textitsampling error w.r.t. the joint on-policy distribution produces inaccurate gradient estimates that can make agents converge sub-optimally. We hypothesize that joint sampling error can be reduced through coordinated action selection and that doing so will increase the reliability of policy gradient learning in MARL (i.e., the probability of converging to an optimal joint policy). To test this hypothesis, we first introduce an adaptive action sampling approach to reduce joint sampling error in the Centralized Training with Decentralized Execution setting. Our method, Cooperative Sampling Error Reduction (CoSER), continually adapts a centralized behavior policy to place higher probability on joint actions that are under-sampled w.r.t. the current joint policy. We then empirically evaluate CoSER on a diverse set of multi-agent games and demonstrate that (1) CoSER reduces joint sampling error more efficiently than independent on-policy sampling and (2) this reduction increases the reliability of independent policy gradient algorithms.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 58 references that are in the map. First posted on arXiv 1 Aug 2025.
Builds on
38 of 58 references.
References · 38
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2015Trust Region Policy Optimizationcited by 2,580
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
- 2017Counterfactual Multi-Agent Policy Gradientscited by 751
- 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
- 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
- 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
- 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 294
26 more
- 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
- 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
- 2019MAVEN: Multi-Agent Variational Explorationcited by 185
- 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 116
- 2018Stochastic Variance-Reduced Policy Gradientcited by 106
- 2019Deep Coordination Graphscited by 98
- 2019Influence-Based Multi-Agent Explorationcited by 75
- 2023Heterogeneous-Agent Reinforcement Learningcited by 69
- 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
- 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 56
- 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
- 2018Policy Optimization via Importance Samplingcited by 47
- 2017Lenient Multi-Agent Deep Reinforcement Learningcited by 35
- 2020UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learningcited by 33
- 2021Active Offline Policy Selectioncited by 26
- 2022PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaborationcited by 23
- 2022ReVar: Strengthening Policy Evaluation via Reduced Variance Samplingcited by 16
- 2018Efficient Counterfactual Learning from Bandit Feedbackcited by 15
- 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
- 2022Safe Exploration for Efficient Policy Evaluation and Comparisoncited by 15
- 2022Pareto Actor-Critic for Equilibrium Selection in Multi-Agent Reinforcement Learningcited by 14
- 2020Reducing Sampling Error in Batch Temporal Difference Learningcited by 11
- 2023Self-Motivated Multi-Agent Explorationcited by 7
- 2022Variance-Optimal Augmentation Logging for Counterfactual Evaluation in Contextual Banditscited by 6
- 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Samplingcited by 6
- 2023Optimistic Multi-Agent Policy Gradientcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning
- 2024Multi-agent cooperation through learning-aware policy gradients
- 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning
- 2026Submodular Multi-Agent Policy Learning for Online Distributed Task Allocation in Open Multi-Agent Systems ↗
- 2025Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
- 2026ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning
- 2026Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning
- 2026Action-Graph Policies: Learning Action Co-dependencies in Multi-Agent Reinforcement Learning