mlsanityData to 7 Oct 2026

Revisiting Bellman Errors for Offline Model Selection

arXiv 2302.00141 ↗First posted 31 Jan 2023cs.LG cs.AI stat.MLcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline model selection (OMS), that is, choosing the best policy from a set of many policies given only logged data, is crucial for applying offline RL in real-world settings. One idea that has been extensively explored is to select policies based on the mean squared Bellman error (MSBE) of the associated Q-functions. However, previous work has struggled to obtain adequate OMS performance with Bellman errors, leading many researchers to abandon the idea. To this end, we elucidate why previous work has seen pessimistic results with Bellman errors and identify conditions under which OMS algorithms based on Bellman errors will perform well. Moreover, we develop a new estimator of the MSBE that is more accurate than prior methods. Our estimator obtains impressive OMS performance on diverse discrete control tasks, including Atari games.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 57 of its 97 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

57 of 97 references.

Closest ancestors · 14

Same direction, newest first

  1. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning3 Nov
  2. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data16 Oct
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  4. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
  5. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  6. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model13 Apr
  7. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  8. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  9. 2020Batch Value-function Approximation with Only Realizability11 Aug
  10. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
31 more
  1. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  2. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  4. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  6. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  8. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  11. 2019Batch Policy Learning under Constraintscited by 242
  12. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  13. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  14. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  15. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  16. 2020BADGR: An Autonomous Self-Supervised Learning-Based Navigation Systemcited by 97
  17. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  18. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  19. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  20. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  21. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  22. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  23. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  24. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  25. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  26. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  27. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
  28. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  29. 2021Minimax Model Learningcited by 16
  30. 2020Causal Effect Estimation and Optimal Dose Suggestions in Mobile Healthcited by 8
  31. 2022Hyperparameter Selection Methods for Fitted Q-Evaluation with Error Guaranteecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 1; 2026: 1.

5 papers.

  1. 2023Towards Robust Multi-Modal Reasoning via Model Selectioncited by 6
  2. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocolcited by 6
  3. 2023When is Offline Policy Selection Sample Efficient for Reinforcement Learning?cited by 3
  4. 2024Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learningcited by 3
  5. 2026Piece of CAKE: Adaptive Execution Engines via Microsecond-Scale Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
  2. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator
  3. 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation
  4. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learning
  5. 2026Residuals-based Offline Reinforcement Learning
  6. 2023LLQL: Logistic Likelihood Q-Learning for Reinforcement Learning
  7. 2024UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning
  8. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data