Accelerating Q-learning through Efficient Value-Sharing across Actions
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Action values are foundational to many control algorithms such as Q-learning. Therefore, efficient action-value learning is central to reinforcement learning (RL). However, learning them can be slow, requiring many updates to move values from their initialization, typically near zero, to their true values, which may be far from zero. Moreover, action-value learning algorithms typically update each state-action pair independently, without learning a value that is common to all actions within a state. In this paper, we address these inefficiencies by introducing the mean-expansion layer, which accelerates action-value learning by sharing values across actions within a state and by changing the problem from directly learning potentially large action-values to learning a lower-norm representation of them. In deep RL, this layer can be applied as a parameter-free addition to Q-network architectures without altering the underlying algorithm. Applied to deep Q-networks and implicit quantile networks, it improves aggregate performance across 57 Atari 2600 games while increasing action gaps and dramatically reducing value overestimation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 39 references that are in the map. First posted on arXiv 29 Jun 2026.
Builds on
19 of 39 references.
Closest ancestors · 2
Same direction, newest first
- 2025Deep Double Q-learning30 Jun
- 2024Reward Centering16 May
Other references · 17
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
- 2017A Distributional Perspective on Reinforcement Learningcited by 765
- 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
- 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
- 2016Learning values across many orders of magnitudecited by 85
- 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
- 2019ChainerRL: A Deep Reinforcement Learning Librarycited by 51
5 more
- 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
- 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
- 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
- 2023VA-learning as a more efficient alternative to Q-learningcited by 11
- 2025An Analysis of Action-Value Temporal-Difference Methods That Learn State Valuescited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning
- 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
- 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
- 2024Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
- 2024Stochastic Q-learning for Large Discrete Action Spaces
- 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Action
- 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
- 2024Mixed Q-Functionals: Advancing Value-Based Methods in Cooperative MARL with Continuous Action Domains