mlsanityData to 14 Oct 2026

A Kernel Loss for Solving the Bellman Equation

arXiv 1905.10506 ↗First posted 25 May 2019cs.LG stat.MLcited by 37

Not on the map. Why?

Abstract (arXiv)

Value function learning plays a central role in many state-of-the-art reinforcement-learning algorithms. Many popular algorithms like Q-learning do not optimize any objective function, but are fixed-point iterations of some variant of Bellman operator that is not necessarily a contraction. As a result, they may easily lose convergence guarantees, as can be observed in practice. In this paper, we propose a novel loss function, which can be optimized using standard gradient-based methods without risking divergence. The key advantage is that its gradient can be easily approximated using sampled transitions, avoiding the need for double samples required by prior algorithms like residual gradient. Our approach may be combined with general function classes such as neural networks, on either on- or off-policy data, and is shown to work reliably and effectively in several benchmarks.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 12; 2022: 9; 2023: 10; 2024: 1; 2025: 3; 2026: 1.

37 papers.

Sort:

By area:

  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  2. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  3. 2021Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approachcited by 44
  4. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  5. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  6. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 32
  7. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  8. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  9. 2021Optimal policy evaluation using kernel-based temporal difference methodscited by 24
  10. 2020Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparisoncited by 22
  11. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21
  12. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 20
25 more
  1. 2021Minimax Model Learningcited by 16
  2. 2021Bayesian Bellman Operatorscited by 16
  3. 2023TD Convergence: An Optimization Perspectivecited by 16
  4. 2021Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settingscited by 14
  5. 2022A Unified Framework for Alternating Offline Model Training and Policy Learningcited by 14
  6. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  7. 2022Inference on Strongly Identified Functionals of Weakly Identified Functionscited by 12
  8. 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11
  9. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  10. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8
  11. 2022Robust Losses for Learning Value Functionscited by 7
  12. 2023Revisiting a Design Choice in Gradient Temporal Difference Learningcited by 7
  13. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  14. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrabilitycited by 5
  15. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  16. 2021Instrument Space Selection for Kernel Maximum Moment Restrictioncited by 2
  17. 2023K-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Controlcited by 2
  18. 2023A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using L-λ Smoothnesscited by 2
  19. 2025Fitted Q-Evaluation without Bellman Completeness via Occupancy Weightingcited by 2
  20. 2022Hyperparameter Selection Methods for Fitted Q-Evaluation with Error Guaranteecited by 1
  21. 2023Neural Network Approximation for Pessimistic Offline Reinforcement Learningcited by 1
  22. 2026A Covering Framework for Offline POMDPs Learning using Belief Space Metriccited by 1
  23. 2022Control of Continuous Quantum Systems with Many Degrees of Freedom based on Convergent Reinforcement Learningcited by 0
  24. 2023LLQL: Logistic Likelihood Q-Learning for Reinforcement Learningcited by 0
  25. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0