# Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics

arXiv: https://arxiv.org/abs/1910.01465 · HTML: https://mlsanity.com/paper/1910.01465/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2019.

## Built on by (most cited first, 30 of 47)

- [MADiff: Offline Multi-agent Learning with Diffusion Models](https://mlsanity.com/paper/2305.17330/) (2023)
- [LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions](https://mlsanity.com/paper/2405.11106/) (2024)
- [Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification](https://mlsanity.com/paper/2111.11188/) (2021)
- [Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks](https://mlsanity.com/paper/2110.14300/) (2021)
- [Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2309.12696/) (2023)
- [Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation](https://mlsanity.com/paper/2211.02127/) (2022)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Regularized Softmax Deep Multi-Agent Q-Learning](https://mlsanity.com/paper/2103.11883/) (2021)
- [Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL](https://mlsanity.com/paper/2307.01472/) (2023)
- [Beyond the Edge: An Advanced Exploration of Reinforcement Learning for Mobile Edge Computing, its Applications, and Future Research Trajectories](https://mlsanity.com/paper/2404.14238/) (2024)
- [Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.00521/) (2023)
- [LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems](https://mlsanity.com/paper/2511.19368/) (2025)
- [Exponential Topology-enabled Scalable Communication in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2502.19717/) (2025)
- [MDPGT: Momentum-based Decentralized Policy Gradient Tracking](https://mlsanity.com/paper/2112.02813/) (2021)
- [Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review](https://mlsanity.com/paper/2208.12328/) (2022)
- [Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2410.08540/) (2024)
- [LEED: A Highly Efficient and Scalable LLM-Empowered Expert Demonstrations Framework for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.14680/) (2025)
- [Stabilizing Voltage in Power Distribution Networks via Multi-Agent Reinforcement Learning with Transformer](https://mlsanity.com/paper/2206.03721/) (2022)
- [Effective Multi-Agent Deep Reinforcement Learning Control with Relative Entropy Regularization](https://mlsanity.com/paper/2309.14727/) (2023)
- [Self-Confirming Transformer for Belief-Conditioned Adaptation in Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.04579/) (2023)
- [Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.12001/) (2024)
- [Decentralized Multi-Agent Reinforcement Learning with Global State Prediction](https://mlsanity.com/paper/2306.12926/) (2023)
- [Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies](https://mlsanity.com/paper/2602.18291/) (2026)
- [Safety Constrained Multi-Agent Reinforcement Learning for Active Voltage Control](https://mlsanity.com/paper/2405.08443/) (2024)
- [Temporal Prototype-Aware Learning for Active Voltage Control on Power Distribution Networks](https://mlsanity.com/paper/2406.17818/) (2024)
- [TRIDENT: Breaking the Hybrid-Safety-Physics Coupling for Provably Safe Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.18308/) (2026)
- [Learning to Play Soccer From Scratch: Sample-Efficient Emergent Coordination through Curriculum-Learning and Competition](https://mlsanity.com/paper/2103.05174/) (2021)
- [Transferable Multi-Agent Reinforcement Learning with Dynamic Participating Agents](https://mlsanity.com/paper/2208.02424/) (2022)
- [Value-Based Deep Multi-Agent Reinforcement Learning with Dynamic Sparse Training](https://mlsanity.com/paper/2409.19391/) (2024)
- [Scalable Safe Multi-Agent Reinforcement Learning for Multi-Agent System](https://mlsanity.com/paper/2501.13727/) (2025)

## Abstract (arXiv)

Many real world tasks require multiple agents to work together. Multi-agent reinforcement learning (RL) methods have been proposed in recent years to solve these tasks, but current methods often fail to efficiently learn policies. We thus investigate the presence of a common weakness in single-agent RL, namely value function overestimation bias, in the multi-agent setting. Based on our findings, we propose an approach that reduces this bias by using double centralized critics. We evaluate it on six mixed cooperative-competitive tasks, showing a significant advantage over current methods. Finally, we investigate the application of multi-agent methods to high-dimensional robotic tasks and show that our approach can be used to learn decentralized policies in this domain.