mlsanityData to 14 Oct 2026

Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics

arXiv 2608.16182 ↗First posted 17 Aug 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable. Existing studies often attribute instability to isolated factors such as overestimation bias or representation learning issues, lacking a unified understanding of how different sources of instability interact during recursive value estimation. In this work, we provide a systematic analysis of instability in deep Q-learning from three complementary perspectives: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under aggressive data reuse. We identify a reward-triggered self-reinforcing trap and characteristic parameter spike dynamics, then derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 39 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

32 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning1 Mar

Other references · 31

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2023Mastering Diverse Domains through World Modelscited by 1,253
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  7. 2016Deep Exploration via Bootstrapped DQNcited by 527
  8. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  10. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  11. 2022Transformers are Sample-Efficient World Modelscited by 328
  12. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
19 more
  1. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  2. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  3. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  4. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  5. 2023Understanding plasticity in neural networkscited by 174
  6. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  7. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  8. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  9. 2019When to use parametric models in reinforcement learning?cited by 118
  10. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  11. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  12. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learningcited by 89
  13. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  14. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  15. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  16. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learningcited by 8
  17. 2024Learning to Play Atari in a World of Tokenscited by 6
  18. 2024Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficientcited by 4
  19. 2025Simple, Good, Fast: Self-Supervised World Models Free of Baggagecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Is Q-learning an Ill-posed Problem?
  2. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  3. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learning
  4. 2025GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning
  5. 2025Deep Double Q-learning
  6. 2024Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks
  7. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL
  8. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression