Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting
Not on the map. Why?
Abstract (arXiv)
Fitted (Q)-evaluation (FQE) is a standard regression-based method for off-policy evaluation, but under distribution shift, value-function realizability alone does not ensure convergence, and existing analyses often require Bellman completeness. We trace this instability to a geometric mismatch: standard FQE projects Bellman targets in the norm induced by the offline distribution, which need not preserve Bellman contraction. We therefore study occupancy-weighted FQE, which changes only the regression weights. Weighting by a target-policy discounted occupancy ratio aligns the projection norm with the target-policy dynamics and restores contraction of the population projected Bellman operator. We derive finite-sample guarantees with estimated occupancy ratios and function-class misspecification, separating finite-iteration, statistical, approximation, and ratio-estimation errors. Exact occupancy weighting removes the need for Bellman completeness; with estimated weights, approximate completeness and value-function realizability reduce sensitivity to ratio-estimation error, with exact realizability yielding higher-order dependence. Combining occupancy-weighted FQE with fitted occupancy-ratio evaluation gives an end-to-end guarantee governed by the complexities and direct approximation errors of the value-function and occupancy-ratio classes. Under coverage, joint realizability of these two classes suffices for consistent estimation without Bellman or critic-side completeness. Controlled experiments illustrate the projection-norm mechanism and the finite-sample tradeoff between contraction and coverage.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 51 of its 91 references that are in the map. First posted on arXiv 29 Dec 2025.
Builds on
51 of 91 references.
References · 51
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
- 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
- 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
- 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
- 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
- 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
- 2019Batch Policy Learning under Constraintscited by 244
- 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
38 more
- 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
- 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
- 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 162
- 2019Orthogonal Statistical Learningcited by 151
- 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
- 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
- 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
- 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
- 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
- 2020Batch Value-function Approximation with Only Realizabilitycited by 104
- 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
- 2022The Role of Coverage in Online Reinforcement Learningcited by 83
- 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
- 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 71
- 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
- 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 56
- 2017Consistent On-Line Off-Policy Evaluationcited by 51
- 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
- 2020Experience Replay with Likelihood-free Importance Weightscited by 42
- 2021An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gapcited by 40
- 2014Approximate Policy Iteration Schemes: A Comparisoncited by 39
- 2019A Kernel Loss for Solving the Bellman Equationcited by 37
- 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
- 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
- 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
- 2020A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Settingcited by 25
- 2015On Convergence of Emphatic Temporal-Difference Learningcited by 21
- 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
- 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
- 2015Emphatic Temporal-Difference Learningcited by 16
- 2021Minimax Model Learningcited by 16
- 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
- 2024Harnessing Density Ratios for Online Reinforcement Learningcited by 15
- 2025Linear Q-Learning Does Not Diverge in L2: Convergence Rates to a Bounded Setcited by 12
- 2022Bellman Residual Orthogonalization for Offline Reinforcement Learningcited by 10
- 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 10
- 2017On Generalized Bellman Equations and Temporal-Difference Learningcited by 8
- 2025Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inferencecited by 7
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.