mlsanityData to 7 Oct 2026

Bellman Residual Orthogonalization for Offline Reinforcement Learning

arXiv 2203.12786 ↗First posted 24 Mar 2022cs.LGcited by 10

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We propose and analyze a reinforcement learning principle that approximates the Bellman equations by enforcing their validity only along an user-defined space of test functions. Focusing on applications to model-free offline RL with function approximation, we exploit this principle to derive confidence intervals for off-policy evaluation, as well as to optimize over policies within a prescribed policy class. We prove an oracle inequality on our policy optimization procedure in terms of a trade-off between the value and uncertainty of an arbitrary comparator policy. Different choices of test function spaces allow us to tackle different problems within a common framework. We characterize the loss of efficiency in moving from on-policy to off-policy data using our procedures, and establish connections to concentrability coefficients studied in past work. We examine in depth the implementation of our methods with linear function approximation, and provide theoretical guarantees with polynomial-time implementations even when Bellman closure does not hold.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 89 references that are in the map. First posted on arXiv 24 Mar 2022.

Builds on

51 of 89 references.

Closest ancestors · 21

Same direction, newest first

  1. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  2. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  3. 2021Pessimistic Model Selection for Offline Deep Reinforcement Learning29 Nov
  4. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
  5. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  7. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  9. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
  10. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  11. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  12. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
9 more
  1. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  2. 2020Batch Value-function Approximation with Only Realizability11 Aug
  3. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  4. 2020Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning7 Jul
  5. 2020Critic Regularized Regression26 Jun
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 30

  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2019Introduction to Online Convex Optimizationcited by 793
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  4. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  5. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  6. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  10. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  11. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  12. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
18 more
  1. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  2. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  3. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  4. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  5. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  6. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  7. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  8. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  9. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  10. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  11. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  12. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  13. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  14. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  15. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  16. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  17. 2021Minimax Model Learningcited by 16
  18. 2021Model Selection in Batch Policy Optimizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 0; 2025: 1; 2026: 2.

10 papers.

  1. 2022On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RLcited by 27
  2. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  3. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  4. 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
  5. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 9
  6. 2023A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learningcited by 9
  7. 2022Policy evaluation from a single path: Multi-step methods, mixing and mis-specificationcited by 3
  8. 2025Fitted Q-Evaluation without Bellman Completeness via Occupancy Weightingcited by 1
  9. 2026A Covering Framework for Offline POMDPs Learning using Belief Space Metriccited by 1
  10. 2026Continuous-time reinforcement learning: ellipticity enables model-free value function approximationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Residuals-based Offline Reinforcement Learning
  2. 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation
  3. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  4. 2026Lifted Bellman Linear Programming for Offline Reinforcement Learning
  5. 2022Learning Bellman Complete Representations for Offline Policy Evaluation
  6. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
  7. 2025Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning
  8. 2025Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood