# Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigation

arXiv: https://arxiv.org/abs/2112.09012 · HTML: https://mlsanity.com/paper/2112.09012/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning](https://mlsanity.com/paper/1709.10082/) (2017)
- [Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments](https://mlsanity.com/paper/1612.05533/) (2016)
- [A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers](https://mlsanity.com/paper/1804.07871/) (2018)
- [Q-Learning in enormous action spaces via amortized approximate maximization](https://mlsanity.com/paper/2001.08116/) (2020)

## Built on by (most cited first, 13 of 13)

- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Distributed Reinforcement Learning for Robot Teams: A Review](https://mlsanity.com/paper/2204.03516/) (2022)
- [BEDI: A Comprehensive Benchmark for Evaluating Embodied Agents on UAVs](https://mlsanity.com/paper/2505.18229/) (2025)
- [Safe Deep Reinforcement Learning by Verifying Task-Level Properties](https://mlsanity.com/paper/2302.10030/) (2023)
- [Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigation](https://mlsanity.com/paper/2302.06695/) (2023)
- [Improving Deep Policy Gradients with Value Function Search](https://mlsanity.com/paper/2302.10145/) (2023)
- [Improving Policy Optimization via varepsilon-Retrain](https://mlsanity.com/paper/2406.08315/) (2024)
- [On Stateful Value Factorization in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2408.15381/) (2024)
- [Safe Multiagent Coordination via Entropic Exploration](https://mlsanity.com/paper/2412.20361/) (2024)
- [Scaling #DNN-Verification Tools with Efficient Bound Propagation and Parallel Computing](https://mlsanity.com/paper/2312.05890/) (2023)
- [Aquatic Navigation: A Challenging Benchmark for Deep Reinforcement Learning](https://mlsanity.com/paper/2405.20534/) (2024)
- [Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges](https://mlsanity.com/paper/2508.06832/) (2025)

## Abstract (arXiv)

We study the problem of multi-robot mapless navigation in the popular Centralized Training and Decentralized Execution (CTDE) paradigm. This problem is challenging when each robot considers its path without explicitly sharing observations with other robots and can lead to non-stationary issues in Deep Reinforcement Learning (DRL). The typical CTDE algorithm factorizes the joint action-value function into individual ones, to favor cooperation and achieve decentralized execution. Such factorization involves constraints (e.g., monotonicity) that limit the emergence of novel behaviors in an individual as each agent is trained starting from a joint action-value. In contrast, we propose a novel architecture for CTDE that uses a centralized state-value network to compute a joint state-value, which is used to inject global state information in the value-based updates of the agents. Consequently, each model computes its gradient update for the weights, considering the overall state of the environment. Our idea follows the insights of Dueling Networks as a separate estimation of the joint state-value has both the advantage of improving sample efficiency, while providing each robot information whether the global state is (or is not) valuable. Experiments in a robotic navigation task with 2 4, and 8 robots, confirm the superior performance of our approach over prior CTDE methods (e.g., VDN, QMIX).