Data to 7 Oct 2026

Diagnosing Bottlenecks in Deep Q-learning Algorithms

arXiv 1902.10250 ↗First posted 26 Feb 2019cs.LG stat.MLcited by 73

Not on the map. Why?

Cited by 34 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Q-learning methods represent a commonly used class of algorithms in reinforcement learning: they are generally efficient and simple, and can be combined readily with function approximators for deep reinforcement learning (RL). However, the behavior of Q-learning methods with function approximation is poorly understood, both theoretically and empirically. In this work, we aim to experimentally investigate potential issues in Q-learning, by means of a "unit testing" framework where we can utilize oracles to disentangle sources of error. Specifically, we investigate questions related to function approximation, sampling error and nonstationarity, and where available, verify if trends found in oracle settings hold true with modern deep RL methods. We find that large neural network architectures have many benefits with regards to learning stability; offer several practical compensations for overfitting; and develop a novel sampling method based on explicitly compensating for function approximation error that yields fair improvement on high-dimensional continuous control domains.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 4; 2021: 19; 2022: 16; 2023: 14; 2024: 9; 2025: 5; 2026: 4.

73 papers.

Sort:

By area:

2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2019Behavior Regularized Offline Reinforcement Learningcited by 654
2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
2021Training Larger Networks for Deep Reinforcement Learningcited by 37

38 more

2023Contrastive Difference Predictive Codingcited by 35
2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
2021Reverb: A Framework For Experience Replaycited by 27
2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
2021Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimizationcited by 22
2022Pareto Conditioned Networkscited by 22
2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
2021Functional Regularization for Reinforcement Learning via Learned Fourier Featurescited by 17
2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17
2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
2025Generative Models in Decision Making: A Surveycited by 14
2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraintscited by 13
2023Boosting Offline Reinforcement Learning with Action Preference Querycited by 12
2025Value Flowscited by 12
2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
2026Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RLcited by 11
2021Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learningcited by 10
2024Learning from Sparse Offline Datasets via Conservative Density Estimationcited by 9
2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
2023Offline RL with Observation Histories: Analyzing and Improving Sample Complexitycited by 7
2021Modularity in Reinforcement Learning via Algorithmic Independence in Credit Assignmentcited by 6
2023Decoupled Prioritized Resampling for Offline RLcited by 6
2019ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrationscited by 5
2021ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectivescited by 5
2021Convergence Results For Q-Learning With Experience Replaycited by 5
2023Mimicking Better by Matching the Approximate Action Distributioncited by 5
2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samplescited by 5
2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learningcited by 5
2021On the Estimation Bias in Double Q-Learningcited by 4
2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RLcited by 4
2022Multi-objective Optimization of Notifications Using Offline Reinforcement Learningcited by 4
2023Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practicecited by 4

50 most cited of 73 All 73 (JSON)