REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes

arXiv 2401.08850 ↗First posted 16 Jan 2024cs.LG cs.AIcited by 6

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Discrete-action reinforcement learning algorithms often falter in tasks with high-dimensional discrete action spaces due to the vast number of possible actions. A recent advancement leverages value-decomposition, a concept from multi-agent reinforcement learning, to tackle this challenge. This study delves deep into the effects of this value-decomposition, revealing that whilst it curtails the over-estimation bias inherent to Q-learning algorithms, it amplifies target variance. To counteract this, we present an ensemble of critics to mitigate target variance. Moreover, we introduce a regularisation loss that helps to mitigate the effects that exploratory actions in one dimension can have on the value of optimal actions in other dimensions. Our novel algorithm, REValueD, tested on discretised versions of the DeepMind Control Suite tasks, showcases superior performance, especially in the challenging humanoid and dog tasks. We further dissect the factors influencing REValueD's performance, evaluating the significance of the regularisation loss and the scalability of REValueD with increasing sub-actions per dimension.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 39 of its 56 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

39 of 56 references.

References · 39

2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2017CARLA: An Open Urban Driving Simulatorcited by 2,449
2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
2017A Distributional Perspective on Reinforcement Learningcited by 765
2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
2016Deep Exploration via Bootstrapped DQNcited by 527
2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
2020dm_control: Software and Tasks for Continuous Controlcited by 375

27 more

2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
2018Distributed Prioritized Experience Replaycited by 269
2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 148
2017Socially Aware Motion Planning with Deep Reinforcement Learningcited by 136
2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
2017Action Branching Architectures for Deep Reinforcement Learningcited by 92
2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 51
2019Measuring the Reliability of Reinforcement Learning Algorithmscited by 51
2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 42
2021Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policiescited by 38
2017Lenient Multi-Agent Deep Reinforcement Learningcited by 35
2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
2022Solving Continuous Control via Q-learningcited by 24
2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 23
2017Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learningcited by 15
2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
2023Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedbackcited by 9
2023Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolutioncited by 7
2024Stochastic Q-learning for Large Discrete Action Spacescited by 5
2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 4
2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
2025Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learningcited by 1
2026Action-Free Offline-to-Online RL via Discretised State Policiescited by 0

Similar, not cited

Close in wording, no citation either way