# Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting

arXiv: https://arxiv.org/abs/2512.23805 · HTML: https://mlsanity.com/paper/2512.23805/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 91 references that are in the map.
- First posted on arXiv 29 Dec 2025.

## Builds on (most cited first, 30 of 51)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Provably Efficient Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/1907.05388/) (2019)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Information-Theoretic Considerations in Batch Reinforcement Learning](https://mlsanity.com/paper/1905.00360/) (2019)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Bellman-consistent Pessimism for Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06926/) (2021)
- [DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections](https://mlsanity.com/paper/1906.04733/) (2019)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation](https://mlsanity.com/paper/1810.12429/) (2018)
- [Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms](https://mlsanity.com/paper/2102.00815/) (2021)
- [Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound](https://mlsanity.com/paper/1905.10389/) (2019)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Orthogonal Statistical Learning](https://mlsanity.com/paper/1901.09036/) (2019)
- [Minimax Weight and Q-Function Learning for Off-Policy Evaluation](https://mlsanity.com/paper/1910.12809/) (2019)
- [What are the Statistical Limits of Offline RL with Linear Function Approximation?](https://mlsanity.com/paper/2010.11895/) (2020)
- [Offline Reinforcement Learning with Realizability and Single-policy Concentrability](https://mlsanity.com/paper/2202.04634/) (2022)
- [Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1911.06854/) (2019)
- [Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes](https://mlsanity.com/paper/1908.08526/) (2019)
- [Batch Value-function Approximation with Only Realizability](https://mlsanity.com/paper/2008.04990/) (2020)
- [An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning](https://mlsanity.com/paper/1503.04269/) (2015)
- [The Role of Coverage in Online Reinforcement Learning](https://mlsanity.com/paper/2210.04157/) (2022)
- [Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning](https://mlsanity.com/paper/1909.05850/) (2019)
- [Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation](https://mlsanity.com/paper/2111.10919/) (2021)
- [Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings](https://mlsanity.com/paper/2107.11003/) (2021)
- [Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency](https://mlsanity.com/paper/2102.02981/) (2021)
- [Consistent On-Line Off-Policy Evaluation](https://mlsanity.com/paper/1702.07121/) (2017)
- [Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift](https://mlsanity.com/paper/1901.09455/) (2019)

## Built on by (most cited first, 2 of 2)

- [Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models](https://mlsanity.com/paper/2512.24407/) (2025)
- [Fitted Occupancy-Ratio Evaluation without Bellman Completeness](https://mlsanity.com/paper/2607.05375/) (2026)

## Abstract (arXiv)

Fitted (Q)-evaluation (FQE) is a standard regression-based method for off-policy evaluation, but under distribution shift, value-function realizability alone does not ensure convergence, and existing analyses often require Bellman completeness. We trace this instability to a geometric mismatch: standard FQE projects Bellman targets in the norm induced by the offline distribution, which need not preserve Bellman contraction. We therefore study occupancy-weighted FQE, which changes only the regression weights. Weighting by a target-policy discounted occupancy ratio aligns the projection norm with the target-policy dynamics and restores contraction of the population projected Bellman operator. We derive finite-sample guarantees with estimated occupancy ratios and function-class misspecification, separating finite-iteration, statistical, approximation, and ratio-estimation errors. Exact occupancy weighting removes the need for Bellman completeness; with estimated weights, approximate completeness and value-function realizability reduce sensitivity to ratio-estimation error, with exact realizability yielding higher-order dependence. Combining occupancy-weighted FQE with fitted occupancy-ratio evaluation gives an end-to-end guarantee governed by the complexities and direct approximation errors of the value-function and occupancy-ratio classes. Under coverage, joint realizability of these two classes suffices for consistent estimation without Bellman or critic-side completeness. Controlled experiments illustrate the projection-norm mechanism and the finite-sample tradeoff between contraction and coverage.