mlsanityData to 14 Oct 2026

Deep Weighted Bellman Residual Minimization for Q^* Estimation

arXiv 2609.34593 ↗First posted 28 Sep 2026cs.LG stat.MLcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms also fits: Diffusion and flow policies for offline RL

also in 1 more
Abstract (arXiv)

Off-policy evaluation is a foundational component of offline reinforcement learning, aiming to assess and optimize policy performance using pre-collected datasets. However, such datasets often suffer from pronounced challenges, including distribution shift, Q-value overestimation, and low sample utilization efficiency. To address these issues, this paper introduces a weighted Bellman residual minimization framework that incorporates density ratio weighting by effectively integrating expert demonstrations with behavioral data. The proposed weighting scheme departs from the conventional completeness assumption commonly imposed in the theoretical analysis of deep reinforcement learning. We establish a sharp convergence rate for density ratio estimation and derive the convergence rate for the excess risk of resulting deep Q^* estimator. Extensive empirical evaluations demonstrate that, compared to existing methods, our method achieves significant improvements in numerical performance and policy generalization, providing specific guidance for the rational utilization of expert demonstrations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 55 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

17 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data14 May
  2. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps25 Mar

Other references · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  5. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  6. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  8. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  9. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  10. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  11. 2016Improved Recurrent Neural Networks for Session-based Recommendationscited by 86
  12. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
3 more
  1. 2022Estimating and Penalizing Induced Preference Shifts in Recommender Systemscited by 29
  2. 2018Active Deep Q-learning with Demonstrationcited by 10
  3. 2021On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correctioncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Residuals-based Offline Reinforcement Learning
  2. 2026Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
  3. 2026Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning
  4. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learning
  5. 2025Stability and Generalization for Bellman Residuals
  6. 2025Bellman Calibration for V-Learning in Offline Reinforcement Learning
  7. 2026Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergence
  8. 2022Learning Bellman Complete Representations for Offline Policy Evaluation