# Fitted Occupancy-Ratio Evaluation without Bellman Completeness

arXiv: https://arxiv.org/abs/2607.05375 · HTML: https://mlsanity.com/paper/2607.05375/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 16 of its 74 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 16 of 16)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [AlgaeDICE: Policy Gradient from Arbitrary Experience](https://mlsanity.com/paper/1912.02074/) (2019)
- [What are the Statistical Limits of Offline RL with Linear Function Approximation?](https://mlsanity.com/paper/2010.11895/) (2020)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation](https://mlsanity.com/paper/2111.10919/) (2021)
- [Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency](https://mlsanity.com/paper/2102.02981/) (2021)
- [A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Setting](https://mlsanity.com/paper/2011.01075/) (2020)
- [LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2202.13536/) (2022)
- [Reinforcement Learning in Low-Rank MDPs with Density Features](https://mlsanity.com/paper/2302.02252/) (2023)
- [Source Condition Double Robust Inference on Functionals of Inverse Problems](https://mlsanity.com/paper/2307.13793/) (2023)
- [Semiparametric Double Reinforcement Learning with Applications to Long-Term Causal Inference](https://mlsanity.com/paper/2501.06926/) (2025)
- [Inverse Reinforcement Learning with Just Classification and a Few Regressions](https://mlsanity.com/paper/2509.21172/) (2025)
- [A Researcher's Guide to Empirical Risk Minimization](https://mlsanity.com/paper/2602.21501/) (2026)
- [Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models](https://mlsanity.com/paper/2512.24407/) (2025)
- [Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting](https://mlsanity.com/paper/2512.23805/) (2025)
- [Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing](https://mlsanity.com/paper/2512.23927/) (2025)

## Built on by (most cited first, 1 of 1)

- [Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning](https://mlsanity.com/paper/2608.24858/) (2026)

## Abstract (arXiv)

Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback-Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to approximation error and a statistical error governed by the complexity of the ratio hypothesis class. When full coverage fails, we introduce coverage-stopped FORE, which targets the discounted occupancy accumulated before the first uncovered state-action pair and yields a conservative lower bound on target-policy value for nonnegative rewards. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.