mlsanityData to 7 Oct 2026

On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluation

arXiv 2201.06169 ↗First posted 17 Jan 2022math.ST cs.LG econ.EMcited by 33

Not on the map. Why?

Abstract (arXiv)

We study the off-policy evaluation (OPE) problem in an infinite-horizon Markov decision process with continuous states and actions. We recast the Q-function estimation into a special form of the nonparametric instrumental variables (NPIV) estimation problem. We first show that under one mild condition the NPIV formulation of Q-function estimation is well-posed in the sense of L2-measure of ill-posedness with respect to the data generating distribution, bypassing a strong assumption on the discount factor γ imposed in the recent literature for obtaining the L2 convergence rates of various Q-function estimators. Thanks to this new well-posed property, we derive the first minimax lower bounds for the convergence rates of nonparametric estimation of Q-function and its derivatives in both sup-norm and L2-norm, which are shown to be the same as those for the classical nonparametric regression (Stone, 1982). We then propose a sieve two-stage least squares estimator and establish its rate-optimality in both norms under some mild conditions. Our general results on the well-posedness and the minimax lower bounds are of independent interest to study not only other nonparametric estimators for Q-function but also efficient estimation on the value of any target policy in off-policy settings.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 26 of its 63 references that are in the map. First posted on arXiv 17 Jan 2022.

Builds on

26 of 63 references.

References · 26

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  3. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2019Batch Policy Learning under Constraintscited by 242
  7. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  8. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  9. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  10. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  11. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  12. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
14 more
  1. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  2. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  3. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  4. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  5. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  6. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  7. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  8. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  9. 2015Optimal Sup-norm Rates and Uniform Inference on Nonlinear Functionals of Nonparametric IV Regressioncited by 30
  10. 2020Statistically Efficient Off-Policy Policy Gradientscited by 24
  11. 2021Optimal policy evaluation using kernel-based temporal difference methodscited by 24
  12. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  13. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 20
  14. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 8; 2024: 6; 2025: 5; 2026: 1.

33 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  3. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  4. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  5. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  6. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  7. 2022Conformal Off-policy Predictioncited by 14
  8. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  9. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
  10. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learningcited by 10
  11. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
  12. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 9
21 more
  1. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  2. 2023Sequential Knockoffs for Variable Selection in Reinforcement Learningcited by 7
  3. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  4. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  5. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  6. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  7. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  8. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  9. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  10. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
  11. 2023STEEL: Singularity-aware Reinforcement Learningcited by 3
  12. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  13. 2023A Reinforcement Learning Framework for Dynamic Mediation Analysiscited by 2
  14. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2
  15. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  16. 2022Inference on Time Series Nonparametric Conditional Moment Restrictions Using General Sievescited by 1
  17. 2024A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimizationcited by 1
  18. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  19. 2025Fitted Q-Evaluation without Bellman Completeness via Occupancy Weightingcited by 1
  20. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  21. 2025The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportabilitycited by 0