mlsanityData to 7 Oct 2026

Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theory

arXiv 2202.04970 ↗First posted 10 Feb 2022stat.ML cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Off-Policy Evaluation (OPE) serves as one of the cornerstones in Reinforcement Learning (RL). Fitted Q Evaluation (FQE) with various function approximators, especially deep neural networks, has gained practical success. While statistical analysis has proved FQE to be minimax-optimal with tabular, linear and several nonparametric function families, its practical performance with more general function approximator is less theoretically understood. We focus on FQE with general differentiable function approximators, making our theory applicable to neural function approximations. We approach this problem using the Z-estimation theory and establish the following results: The FQE estimation error is asymptotically normal with explicit variance determined jointly by the tangent space of the function class at the ground truth, the reward structure, and the distribution shift due to off-policy learning; The finite-sample FQE error bound is dominated by the same variance term, and it can also be bounded by function class-dependent divergence, which measures how the off-policy distribution shift intertwines with the function approximator. In addition, we study bootstrapping FQE estimators for error distribution inference and estimating confidence intervals, accompanied by a Cramer-Rao lower bound that matches our upper bounds. The Z-estimation analysis provides a generalizable theoretical framework for studying off-policy estimation in RL and provides sharp statistical theory for FQE with differentiable function approximators.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 10 Feb 2022.

Builds on

28 of 48 references.

References · 28

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  4. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  5. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  6. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  7. 2019Batch Policy Learning under Constraintscited by 242
  8. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  9. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  10. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  11. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  12. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
16 more
  1. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  2. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  3. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  4. 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
  5. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  6. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  7. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  8. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  9. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  10. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  11. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  12. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  13. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
  14. 2021Optimal policy evaluation using kernel-based temporal difference methodscited by 24
  15. 2020On Function Approximation in Reinforcement Learning: Optimism in the Face of Large State Spacescited by 19
  16. 2021An L2 Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 5; 2024: 1; 2025: 2; 2026: 4.

18 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14
  3. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  4. 2022Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficientcited by 12
  5. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  6. 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
  7. 2023Logarithmic Switching Cost in Reinforcement Learning beyond Linear MDPscited by 8
  8. 2023Non-stationary Reinforcement Learning under General Function Approximationcited by 5
  9. 2022Offline Stochastic Shortest Path: Learning, Evaluation and Towards Optimalitycited by 3
  10. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Datacited by 0
  11. 2023Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networkscited by 0
  12. 2024Is Offline Decision Making Possible with Only Few Samples? Reliable Decisions in Data-Starved Bandits via Trust Region Enhancementcited by 0
6 more
  1. 2025Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Spacecited by 0
  2. 2025Realistic CDSS Drug Dosing with End-to-end Recurrent Q-learning for Dual Vasopressor Controlcited by 0
  3. 2026Formulating Reinforcement Learning for Human-Robot Collaboration through Off-Policy Evaluationcited by 0
  4. 2026Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluationcited by 0
  5. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0
  6. 2026Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learningcited by 0