mlsanityData to 7 Oct 2026

Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error

arXiv 2201.12417 ↗First posted 28 Jan 2022cs.LG cs.AI stat.MLcited by 31

Not on the map. Why?

Abstract (arXiv)

In this work, we study the use of the Bellman equation as a surrogate objective for value prediction accuracy. While the Bellman equation is uniquely solved by the true value function over all state-action pairs, we find that the Bellman error (the difference between both sides of the equation) is a poor proxy for the accuracy of the value function. In particular, we show that (1) due to cancellations from both sides of the Bellman equation, the magnitude of the Bellman error is only weakly related to the distance to the true value function, even when considering all state-action pairs, and (2) in the finite data regime, the Bellman equation can be satisfied exactly by infinitely many suboptimal solutions. This means that the Bellman error can be minimized without improving the accuracy of the value function. We demonstrate these phenomena through a series of propositions, illustrative toy examples, and empirical analysis in standard benchmark domains.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 33 of its 61 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

33 of 61 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
21 more
  1. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  2. 2019Batch Policy Learning under Constraintscited by 242
  3. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  4. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  5. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  6. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  7. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  8. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  9. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  10. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  11. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  12. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  13. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  14. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  15. 2020Logistic Q-Learningcited by 34
  16. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  17. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  18. 2019Deep Residual Reinforcement Learningcited by 21
  19. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  20. 2021Infinite-Horizon Offline Reinforcement Learning with Linear Function Approximation: Curse of Dimensionality and Algorithmcited by 10
  21. 2019Borrowing From the Future: An Attempt to Address Double Samplingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 9; 2024: 5; 2025: 7; 2026: 5.

31 papers.

Sort:

By area:

  1. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  2. 2022Tutorial on amortized optimizationcited by 83
  3. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  4. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  5. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  6. 2022Disentangling Transfer in Continual Reinforcement Learningcited by 31
  7. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14
  8. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 12
  9. 2025Value Flowscited by 12
  10. 2024Koopman-Assisted Reinforcement Learningcited by 11
  11. 2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learningcited by 10
  12. 2024Finding good policies in average-reward Markov Decision Processes without prior knowledgecited by 10
19 more
  1. 2022Learning to Mitigate AI Collusion on Economic Platformscited by 9
  2. 2023Graph Reinforcement Learning for Network Control via Bi-Level Optimizationcited by 7
  3. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocolcited by 6
  4. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  5. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 5
  6. 2025Ego-Vision World Model for Humanoid Contact Planningcited by 5
  7. 2023Learning from Integral Losses in Physics Informed Neural Networkscited by 4
  8. 2025β-DQN: Improving Deep Q-Learning By Evolving the Behaviorcited by 3
  9. 2022An Experimental Comparison Between Temporal Difference and Residual Gradient with Neural Network Approximationcited by 2
  10. 2023Distillation Policy Optimizationcited by 1
  11. 2025Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learningcited by 1
  12. 2026Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergencecited by 1
  13. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 1
  14. 2023LLQL: Logistic Likelihood Q-Learning for Reinforcement Learningcited by 0
  15. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improvecited by 0
  16. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  17. 2026Multi-Agent Model-Based Reinforcement Learning with Joint State-Action Learned Embeddingscited by 0
  18. 2026Contraction-Aligned Analysis of Soft Bellman Residual Minimization with Weighted Lp-Norm for Markov Decision Problemcited by 0
  19. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0