An Advantage-based Optimization Method for Reinforcement Learning in Large Action Space
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Reinforcement learning tasks in real-world scenarios often involve large, high-dimensional action spaces, leading to challenges such as convergence difficulties, instability, and high computational complexity. It is widely acknowledged that traditional value-based reinforcement learning algorithms struggle to address these issues effectively. A prevalent approach involves generating independent sub-actions within each dimension of the action space. However, this method introduces bias, hindering the learning of optimal policies. In this paper, we propose an advantage-based optimization method and an algorithm named Advantage Branching Dueling Q-network (ABQ). ABQ incorporates a baseline mechanism to tune the action value of each dimension, leveraging the advantage relationship across different sub-actions. With this approach, the learned policy can be optimized for each dimension. Empirical results demonstrate that ABQ outperforms BDQ, achieving 3%, 171%, and 84% more cumulative rewards in HalfCheetah, Ant, and Humanoid environments, respectively. Furthermore, ABQ exhibits competitive performance when compared against two continuous action benchmark algorithms, DDPG and TD3.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 17 of its 36 references that are in the map. First posted on arXiv 17 Dec 2024.
Builds on
17 of 36 references.
References · 17
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
- 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
- 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 261
- 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 194
- 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 184
- 2017Action Branching Architectures for Deep Reinforcement Learningcited by 92
- 2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 51
- 2018Managing engineering systems with large state and action spaces through deep reinforcement learningcited by 32
5 more
- 2019Graph Policy Gradients for Large Scale Robot Controlcited by 28
- 2021A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directionscited by 21
- 2020Vehicular Cooperative Perception Through Action Branching and Federated Reinforcement Learningcited by 9
- 2018Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradientscited by 2
- 2022PROMPT: Learning Dynamic Resource Allocation Policies for Network Applicationscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
- 2022An Improved Reinforcement Learning Algorithm for Learning to Branch
- 2026ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
- 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
- 2026Reinforcement Learning via Advantage Guidance
- 2025Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach
- 2026Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
- 2024Time-Scale Separation in Q-Learning: Extending TD(triangle) for Action-Value Function Decomposition