mlsanityData to 14 Oct 2026

Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning

arXiv 2608.24858 ↗First posted 25 Aug 2026cs.LG stat.MLcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characterized by an adjoint Bellman equation. Existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations because of function-class approximation, regularization, or incomplete optimization. These violations are difficult to diagnose and reduce because the objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping. We introduce isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method that reduces these violations while preserving the ranking information in any initial occupancy-ratio estimate. The method corrects the estimate's scale and shape by applying fitted occupancy-ratio evaluation (FORE) over a one-dimensional class of nondecreasing transformations. We characterize Bellman calibration as a conditional fixed-point property equivalent to occupancy-balance against every test function of the calibrated ratio. More generally, we derive a calibration-refinement bound showing that any fitted ratio with small calibration error performs nearly as well as the best post-processing based on its fitted values. For isotonic Bellman calibration, we establish finite-sample calibration guarantees and a KL oracle inequality relative to the best monotone transformation of the initial estimate. Consequently, isotonic Bellman calibration achieves small calibration error and KL risk within statistical error of the best monotone correction, with guarantees for downstream target-occupancy functionals, including policy-value estimation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 87 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

51 of 87 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Fitted Occupancy-Ratio Evaluation without Bellman Completeness6 Jul
  2. 2025Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing30 Dec
  3. 2025Bellman Calibration for V-Learning in Offline Reinforcement Learning29 Dec
  4. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluation30 Nov
  5. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 45

  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  4. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  6. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  7. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  8. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  11. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  12. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
33 more
  1. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  2. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  3. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  4. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  5. 2019Doubly robust off-policy evaluation with shrinkagecited by 87
  6. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  7. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  8. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  9. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  10. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  11. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  12. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  13. 2021Distribution-free calibration guarantees for histogram binning without sample splittingcited by 45
  14. 2019Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learningcited by 28
  15. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  16. 2020Stable discovery of interpretable subgroups via calibration in causal studiescited by 18
  17. 2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 18
  18. 2023Causal isotonic calibration for heterogeneous treatment effectscited by 16
  19. 2021Truncated Emphatic Temporal Difference Methods for Prediction and Controlcited by 10
  20. 2024Doubly robust inference via calibrationcited by 9
  21. 2024Stabilized Inverse Probability Weighting via Isotonic Calibrationcited by 9
  22. 2020Propensity score weighting under limited overlap and model misspecificationcited by 8
  23. 2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapcited by 8
  24. 2025Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inferencecited by 7
  25. 2025Generalized Venn and Venn-Abers Calibration with Applications in Conformal Predictioncited by 7
  26. 2023The Scope of Multicalibration: Characterizing Multicalibration via Property Elicitationcited by 6
  27. 2024Orthogonal Causal Calibrationcited by 6
  28. 2024Improving the Finite Sample Estimation of Average Treatment Effects using Double/Debiased Machine Learning with Propensity Score Calibrationcited by 5
  29. 2020Confidence intervals for multiple isotonic regression and other monotone modelscited by 3
  30. 2023Calibrated and Conformal Propensity Scores for Causal Effect Estimationcited by 3
  31. 2025Calibration Strategies for Robust Causal Estimation: Theoretical and Empirical Insights on Propensity Score-Based Estimatorscited by 3
  32. 2025Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Modelscited by 3
  33. 2025Average-DICE: Stationary Distribution Correction by Regressioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Deep Weighted Bellman Residual Minimization for Q^* Estimation
  2. 2021Projected State-action Balancing Weights for Offline Reinforcement Learning
  3. 2026Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
  4. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
  5. 2026Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning
  7. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
  8. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning