mlsanityData to 7 Oct 2026

Universal Off-Policy Evaluation

arXiv 2104.12820 ↗First posted 26 Apr 2021cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

When faced with sequential decision-making problems, it is often useful to be able to predict what would happen if decisions were made using a new policy. Those predictions must often be based on data collected under some previously used decision-making rule. Many previous methods enable such off-policy (or counterfactual) estimation of the expected value of a performance measure called the return. In this paper, we take the first steps towards a universal off-policy estimator (UnO) -- one that provides off-policy estimates and high-confidence bounds for any parameter of the return distribution. We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns. Finally, we also discuss Uno's applicability in various settings, including fully observable, partially observable (i.e., with unobserved confounders), Markovian, non-Markovian, stationary, smoothly non-stationary, and discrete distribution shifts.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 42 of its 117 references that are in the map. First posted on arXiv 26 Apr 2021.

Builds on

42 of 117 references.

References · 42

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  4. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  6. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  7. 2019Challenges of Real-World Reinforcement Learningcited by 282
  8. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  9. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  10. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  11. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  12. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
30 more
  1. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  2. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  3. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  4. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  5. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  6. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  7. 2020A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environmentscited by 62
  8. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  9. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  10. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  11. 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 42
  12. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  13. 2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
  14. 2020Optimizing for the Future in Non-Stationary MDPscited by 37
  15. 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 36
  16. 2018Importance Sampling Policy Evaluation with an Estimated Behavior Policycited by 34
  17. 2020Offline Policy Selection under Uncertaintycited by 31
  18. 2021Off-Policy Risk Assessment in Contextual Banditscited by 29
  19. 2020Policy Evaluation Networkscited by 28
  20. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  21. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  22. 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
  23. 2020Parameter-Based Value Functionscited by 24
  24. 2018Best Arm Identification for Contaminated Banditscited by 23
  25. 2020Minimax Value Interval for Off-Policy Evaluation and Policy Optimizationcited by 11
  26. 2020Reducing Sampling Error in Batch Temporal Difference Learningcited by 11
  27. 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11
  28. 2020Reinforcement Learning for Strategic Recommendationscited by 8
  29. 2021High-Confidence Off-Policy (or Counterfactual) Variance Estimationcited by 8
  30. 2016Importance Sampling with Unequal Supportcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 12; 2023: 14; 2024: 3; 2025: 3; 2026: 4.

43 papers.

Sort:

By area:

  1. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  2. 2022Anytime-valid off-policy inference for contextual banditscited by 37
  3. 2022Conformal Off-Policy Prediction in Contextual Banditscited by 25
  4. 2023Distributional Soft Actor-Critic with Three Refinementscited by 22
  5. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  6. 2022Conformal Off-policy Predictioncited by 14
  7. 2022Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictionscited by 14
  8. 2021Reimagining an autonomous vehiclecited by 11
  9. 2022The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learningcited by 9
  10. 2021Estimating Potential Outcome Distributions with Collaborating Causal Networkscited by 8
  11. 2023Estimation and Inference in Distributional Reinforcement Learningcited by 8
  12. 2022Enforcing Delayed-Impact Fairness Guaranteescited by 7
31 more
  1. 2022Off-Policy Risk Assessment in Markov Decision Processescited by 7
  2. 2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapcited by 7
  3. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  4. 2021SOPE: Spectrum of Off-Policy Estimatorscited by 5
  5. 2021ESCADA: Efficient Safety and Context Aware Dose Allocation for Precision Medicinecited by 5
  6. 2022A Reinforcement Learning Approach to Estimating Long-term Treatment Effectscited by 5
  7. 2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5
  8. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  9. 2022Opportunities and Challenges from Using Animal Videos in Reinforcement Learning for Navigationcited by 4
  10. 2025Rule-Guided Reinforcement Learning Policy Evaluation and Improvementcited by 4
  11. 2022Off-Policy Evaluation with Policy-Dependent Optimization Responsecited by 3
  12. 2023Probabilistic Offline Policy Ranking with Approximate Bayesian Computationcited by 3
  13. 2025Doubly-Robust Estimation of Counterfactual Policy Mean Embeddingscited by 3
  14. 2022On the Reuse Bias in Off-Policy Reinforcement Learningcited by 2
  15. 2023Off-Policy Evaluation with Out-of-Sample Guaranteescited by 2
  16. 2023Off-Policy Evaluation for Action-Dependent Non-Stationary Environmentscited by 2
  17. 2023Time-uniform confidence bands for the CDF under nonstationaritycited by 2
  18. 2023A Distribution Optimization Framework for Confidence Bounds of Risk Measurescited by 2
  19. 2023A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using L-λ Smoothnesscited by 2
  20. 2023Distributional Off-Policy Evaluation for Slate Recommendationscited by 2
  21. 2024Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learningcited by 2
  22. 2021A Unified Off-Policy Evaluation Approach for General Value Functioncited by 1
  23. 2023Asymptotically Unbiased Off-Policy Policy Evaluation when Reusing Old Data in Nonstationary Environmentscited by 1
  24. 2026Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learningcited by 1
  25. 2021An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processescited by 0
  26. 2022Bayesian Counterfactual Mean Embeddings and Off-Policy Evaluationcited by 0
  27. 2023Conservative Exploration for Policy Optimization via Off-Policy Policy Evaluationcited by 0
  28. 2025Counterfactual Inference under Thompson Samplingcited by 0
  29. 2026Pessimistic Risk-Aware Policy Learning in Contextual Banditscited by 0
  30. 2026Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlapcited by 0
  31. 2026Tail-Influence Sampling for CVaR Policy Evaluationcited by 0