mlsanityData to 7 Oct 2026

Revisiting TD Target Aggregation under Uncertainty in Q-Learning

arXiv 2608.03069 ↗First posted 4 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy maximization over next-state action values. While effective, this aggregation rule is inherently sensitive to estimation noise: when Q-values are uncertain, the maximization operator deterministically favors the largest estimate, regardless of its reliability, leading to amplified errors through bootstrapping. In this work, we propose the Successor Rollout Aggregation Deep Q-Network (SADQ), a simple modification to Q-learning that regularizes how the TD target is formed. SADQ uses one-step rollout predictions from a learned dynamics model to guide the comparison among candidate next-state actions, introducing additional structure into the aggregation step without altering the underlying learning framework. The resulting mixed Bellman update attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. We provide theoretical analysis showing that SADQ reduces bootstrap-induced overestimation in a pointwise manner. Empirically, SADQ consistently improves training stability across classical control tasks, real-world vector-based environments, and Atari benchmarks when compared to strong DQN variants.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 60 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

45 of 60 references.

References · 45

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  12. 2016Deep Exploration via Bootstrapped DQNcited by 518
33 more
  1. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  2. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  3. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  4. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  5. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  6. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  7. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  8. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  9. 2019CityFlow: A City-Scale Benchmark for Multi-Target Multi-Camera Vehicle Tracking and Re-Identificationcited by 119
  10. 2023Synthetic Experience Replaycited by 100
  11. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  12. 2024Simplifying Deep Temporal Difference Learningcited by 71
  13. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  14. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
  15. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 40
  16. 2020Gradient Temporal-Difference Learning with Regularized Correctionscited by 38
  17. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  18. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  19. 2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21
  20. 2019Target-Based Temporal Difference Learningcited by 20
  21. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 17
  22. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  23. 2017A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learningcited by 16
  24. 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
  25. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  26. 2023Reinforcement Learning with Simple Sequence Priorscited by 11
  27. 2023Efficient Dynamics Modeling in Interactive Environments with Koopman Theorycited by 10
  28. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 9
  29. 2024SF-DQN: Provable Knowledge Transfer using Successor Feature for Deep Reinforcement Learningcited by 5
  30. 2022Revisiting Model-based Value Expansioncited by 3
  31. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
  32. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  33. 2026Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learningcited by 1

Built on by

Not cited yet.