mlsanityData to 7 Oct 2026

Bellman Calibration for V-Learning in Offline Reinforcement Learning

arXiv 2512.23694 ↗First posted 29 Dec 2025stat.ML cs.LG econ.EMcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reliable long-horizon value prediction is difficult in offline reinforcement learning because fitted value methods combine bootstrapping, function approximation, and distribution shift, while standard guarantees often require Bellman completeness or realizability. We introduce Bellman calibration, a weak reliability criterion requiring that states assigned similar predicted values have average Bellman targets that agree with those predictions. This criterion yields a scalar calibration error for diagnosing systematic numerical miscalibration, which we estimate from off-policy data using doubly robust Bellman target estimates. We then propose Iterated Bellman Calibration, a model-agnostic post-hoc procedure that recalibrates any learned value predictor by fitting a one-dimensional map of its original prediction, with histogram and isotonic variants. We prove finite-sample guarantees showing that Bellman calibration error is controlled at one-dimensional nonparametric rates without Bellman completeness or value-function realizability. Our value-error bounds separate statistical estimation, finite-iteration, and approximation errors, clarifying when calibration improves value prediction and when its gains are limited by the information in the original predictor or insufficient coverage.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 108 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

54 of 108 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning2 Oct
  2. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage5 Feb
  3. 2022The Role of Coverage in Online Reinforcement Learning9 Oct
  4. 2022Learning Bellman Complete Representations for Offline Policy Evaluation12 Jul
  5. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  8. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  9. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  10. 2021Fast Rates for the Regret of Offline Reinforcement Learning31 Jan
  11. 2020Batch Value-function Approximation with Only Realizability11 Aug
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun

Other references · 42

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2016Progressive Neural Networkscited by 1,332
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  9. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  12. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
30 more
  1. 2019Orthogonal Statistical Learningcited by 150
  2. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  3. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  4. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  5. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  6. 2020Distribution-free binary classification: prediction sets, confidence intervals and calibrationcited by 79
  7. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  8. 2021Distribution-free calibration guarantees for histogram binning without sample splittingcited by 44
  9. 2021An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gapcited by 40
  10. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  11. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  12. 2020Sample Efficient Reinforcement Learning via Low-Rank Matrix Estimationcited by 25
  13. 2020A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Settingcited by 24
  14. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  15. 2015Emphatic Temporal-Difference Learningcited by 16
  16. 2023Causal isotonic calibration for heterogeneous treatment effectscited by 16
  17. 2024Self-Calibrating Conformal Predictioncited by 16
  18. 2022Conformal Off-policy Predictioncited by 14
  19. 2024Bellman Conformal Inference: Calibrating Prediction Intervals For Time Seriescited by 12
  20. 2024Doubly robust inference via calibrationcited by 8
  21. 2024Stabilized Inverse Probability Weighting via Isotonic Calibrationcited by 8
  22. 2023The Scope of Multicalibration: Characterizing Multicalibration via Property Elicitationcited by 6
  23. 2025Generalized Venn and Venn-Abers Calibration with Applications in Conformal Predictioncited by 6
  24. 2022Calibration Error for Heterogeneous Treatment Effectscited by 5
  25. 2023AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systemscited by 5
  26. 2024Orthogonal Causal Calibrationcited by 5
  27. 2025Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inferencecited by 5
  28. 2025Nonparametric Instrumental Variable Inference with Many Weak Instrumentscited by 5
  29. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressionscited by 4
  30. 2020Confidence intervals for multiple isotonic regression and other monotone modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Modelscited by 1
  2. 2026Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning
  2. 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation
  3. 2025Stability and Generalization for Bellman Residuals
  4. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learning
  5. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
  6. 2026Optimal Value Inference for Reinforcement Learning
  7. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  8. 2026Lifted Bellman Linear Programming for Offline Reinforcement Learning