mlsanityData to 7 Oct 2026

Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL

arXiv 2310.04411 ↗First posted 6 Oct 2023cs.LGcited by 20

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The divergence of the Q-value estimation has been a prominent issue in offline RL, where the agent has no access to real dynamics. Traditional beliefs attribute this instability to querying out-of-distribution actions when bootstrapping value targets. Though this issue can be alleviated with policy constraints or conservative Q estimation, a theoretical understanding of the underlying mechanism causing the divergence has been absent. In this work, we aim to thoroughly comprehend this mechanism and attain an improved solution. We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. Then, we propose a novel Self-Excite Eigenvalue Measure (SEEM) metric based on Neural Tangent Kernel (NTK) to measure the evolving property of Q-network at training, which provides an intriguing explanation of the emergence of divergence. For the first time, our theory can reliably decide whether the training will diverge at an early stage, and even predict the order of the growth for the estimated Q-value, the model's norm, and the crashing step when an SGD optimizer is used. The experiments demonstrate perfect alignment with this theoretic analysis. Building on our insights, we propose to resolve divergence from a novel perspective, namely improving the model's architecture for better extrapolating behavior. Through extensive empirical studies, we identify LayerNorm as a good solution to effectively avoid divergence without introducing detrimental bias, leading to superior performance. Experimental results prove that it can still work in some most challenging settings, i.e. using only 1 transitions of the dataset, where all previous methods fail. Moreover, it can be easily plugged into modern offline RL methods and achieve SOTA results on many challenging tasks. We also give unique insights into its effectiveness.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 61 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

50 of 61 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance4 Sep
  2. 2023Decoupled Prioritized Resampling for Offline RL8 Jun
  3. 2023Boosting Offline Reinforcement Learning with Action Preference Query6 Jun
  4. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms1 Jun
  5. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  6. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
  7. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  8. 2022Mutual Information Regularized Offline Reinforcement Learning14 Oct
  9. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  10. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  11. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
8 more
  1. 2021Offline RL Without Off-Policy Evaluation16 Jun
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2020Critic Regularized Regression26 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 30

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
18 more
  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  2. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  3. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  5. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  6. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  7. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  8. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  9. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  10. 2023Adaptive Rotated Convolution for Rotated Object Detectioncited by 51
  11. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  12. 2022Learning to Weight Samples for Dynamic Early-exiting Networkscited by 44
  13. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  14. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  15. 2021On the Power of Multitask Representation Learning in Linear MDPcited by 30
  16. 2022Value-Consistent Representation Learning for Data-Efficient Reinforcement Learningcited by 15
  17. 2021CAM-loss: Towards Learning Spatially Discriminative Feature Representationscited by 12
  18. 2022Provable General Function Class Representation Learning in Multitask Bandits and MDPscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 6; 2026: 4.

20 papers.

Sort:

By area:

  1. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  2. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  3. 2024Simplifying Deep Temporal Difference Learningcited by 71
  4. 2024Real-world Image Dehazing with Coherence-based Pseudo Labeling and Cooperative Unfolding Networkcited by 26
  5. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  6. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  7. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  8. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
  9. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  10. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  11. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
  12. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
8 more
  1. 2024Strategically Conservative Q-Learningcited by 1
  2. 2025Universal Value-Function Uncertaintiescited by 1
  3. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
  4. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  5. 2024Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feedcited by 0
  6. 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Networkcited by 0
  7. 2026Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentationcited by 0
  8. 2026Rank Collapse Is Recoverable, Growing |Q| Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Criticcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  2. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  4. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  5. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
  6. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  7. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  8. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model