mlsanityData to 7 Oct 2026

Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributions

arXiv 2210.15543 ↗First posted 27 Oct 2022cs.LG cs.AI stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

Off-policy evaluation often refers to two related tasks: estimating the expected return of a policy and estimating its value function (or other functions of interest, such as density ratios). While recent works on marginalized importance sampling (MIS) show that the former can enjoy provable guarantees under realizable function approximation, the latter is only known to be feasible under much stronger assumptions such as prohibitively expressive discriminators. In this work, we provide guarantees for off-policy function estimation under only realizability, by imposing proper regularization on the MIS objectives. Compared to commonly used regularization in MIS, our regularizer is much more flexible and can account for an arbitrary user-specified distribution, under which the learned function will be close to the groundtruth. We provide exact characterization of the optimal dual solution that needs to be realized by the discriminator class, which determines the data-coverage assumption in the case of value-function learning. As another surprising observation, the regularizer can be altered to relax the data-coverage requirement, and completely eliminate it in the ideal case with strong side information.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

22 of 42 references.

References · 22

  1. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  2. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  3. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  4. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2019Batch Policy Learning under Constraintscited by 242
  8. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  9. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  10. 2017A Rewriting System for Convex Optimization Problemscited by 178
  11. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  12. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
10 more
  1. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  2. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  3. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  4. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  5. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  6. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  7. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  8. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  9. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  10. 2020Generalization Bounds for Stochastic Saddle Point Problemscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 1; 2025: 0; 2026: 1.

8 papers.

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 17
  3. 2023Minimax Instrumental Variable Regression and L2 Convergence Guarantees without Identification or Closednesscited by 13
  4. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 9
  5. 2023The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimationcited by 3
  6. 2023Offline Reinforcement Learning with Additional Covering Distributionscited by 1
  7. 2026A Unifying View of Coverage in Linear Off-Policy Evaluationcited by 1
  8. 2024A Fast Convergence Theory for Offline Decision Makingcited by 0