mlsanityData to 7 Oct 2026

A Review of Off-Policy Evaluation in Reinforcement Learning

arXiv 2212.06355 ↗First posted 13 Dec 2022stat.ML cs.LG math.STcited by 92

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) is one of the most vibrant research frontiers in machine learning and has been recently applied to solve a number of challenging problems. In this paper, we primarily focus on off-policy evaluation (OPE), one of the most fundamental topics in RL. In recent years, a number of OPE methods have been developed in the statistics and computer science literature. We provide a discussion on the efficiency bound of OPE, some of the existing state-of-the-art OPE methods, their statistical properties and some other related research directions that are currently actively explored.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 95 of its 220 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

95 of 220 references.

References · 95

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  6. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  10. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  11. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  12. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
38 more
  1. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  2. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  3. 2019Orthogonal Statistical Learningcited by 150
  4. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  5. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  6. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  7. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 138
  8. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  9. 2022Semiparametric doubly robust targeted double machine learning: a reviewcited by 129
  10. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  11. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  12. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  13. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  14. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  15. 2019Doubly robust off-policy evaluation with shrinkagecited by 87
  16. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  17. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  18. 2019Deep ReLU network approximation of functions on a manifoldcited by 73
  19. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  20. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  21. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  22. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  23. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  24. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  25. 2015Nonparametric methods for doubly robust estimation of continuous treatment effectscited by 62
  26. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  27. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  28. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  29. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  30. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  31. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  32. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  33. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  34. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  35. 2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
  36. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  37. 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 36
  38. 2020POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learningcited by 36

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 25; 2025: 28; 2026: 24.

92 papers.

Sort:

By area:

  1. 2024Improving Agent Behaviors with RL Fine-tuning for Autonomous Drivingcited by 27
  2. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  3. 2024Model-agnostic meta-learners for estimating heterogeneous treatment effects over timecited by 18
  4. 2023Off-Policy Evaluation for Large Action Spaces via Policy Convolutioncited by 14
  5. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  6. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  7. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  8. 2024IGC-Net for conditional average potential outcome estimation over timecited by 10
  9. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 10
  10. 2024Long-term Off-Policy Evaluation and Learningcited by 9
  11. 2023Improving Behavioural Cloning with Positive Unlabeled Learningcited by 8
  12. 2023Learning Regularized Monotone Graphon Mean-Field Gamescited by 8
38 more
  1. 2024DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regimecited by 8
  2. 2025G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibrationcited by 8
  3. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  4. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  5. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPscited by 7
  6. 2023Conformal Off-Policy Evaluation in Markov Decision Processescited by 6
  7. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  8. 2023Positivity-free Policy Learning with Observational Datacited by 6
  9. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  10. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyondcited by 6
  11. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  12. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  13. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  14. 2024RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluationcited by 5
  15. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  16. 2024Counterfactual Influence in Markov Decision Processescited by 4
  17. 2025Rule-Guided Reinforcement Learning Policy Evaluation and Improvementcited by 4
  18. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  19. 2026A Statistical Framework for Alignment with Biased AI Feedbackcited by 4
  20. 2024Conformal Off-Policy Prediction for Multi-Agent Systemscited by 3
  21. 2024Medical Knowledge Integration into Reinforcement Learning Algorithms for Dynamic Treatment Regimescited by 3
  22. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  23. 2023K-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Controlcited by 2
  24. 2024Predictive Performance Comparison of Decision Policies Under Confoundingcited by 2
  25. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2
  26. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  27. 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatmentcited by 2
  28. 2024Tractable Offline Learning of Regular Decision Processescited by 2
  29. 2024Test Where Decisions Matter: Importance-driven Testing for Deep Reinforcement Learningcited by 2
  30. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  31. 2025STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluationcited by 2
  32. 2025Off-Policy Evaluation and Learning for Matching Marketscited by 2
  33. 2025A Two-armed Bandit Framework for A/B Testingcited by 2
  34. 2025Breaking Determinism: Stochastic Modeling for Reliable Off-Policy Evaluation in Ad Auctionscited by 2
  35. 2024Deep reinforcement learning for tracking a moving target in jellyfish-like swimmingcited by 1
  36. 2024Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluationcited by 1
  37. 2025A Graphical Approach to State Variable Selection in Off-policy Learningcited by 1
  38. 2025Policy Testing in Markov Decision Processescited by 1

50 most cited of 92 All 92 (JSON)