mlsanityData to 14 Oct 2026

Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation

arXiv 2311.18207 ↗First posted 30 Nov 2023cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Off-Policy Evaluation (OPE) aims to assess the effectiveness of counterfactual policies using only offline logged data and is often used to identify the top-k promising policies for deployment in online A/B tests. Existing evaluation metrics for OPE estimators primarily focus on the "accuracy" of OPE or that of downstream policy selection, neglecting risk-return tradeoff in the subsequent online policy deployment. To address this issue, we draw inspiration from portfolio evaluation in finance and develop a new metric, called SharpeRatio@k, which measures the risk-return tradeoff of policy portfolios formed by an OPE estimator under varying online evaluation budgets (k). We validate our metric in two example scenarios, demonstrating its ability to effectively distinguish between low-risk and high-risk estimators and to accurately identify the most efficient one. Efficiency of an estimator is characterized by its capability to form the most advantageous policy portfolios, maximizing returns while minimizing risks during online deployment, a nuance that existing metrics typically overlook. To facilitate a quick, accurate, and consistent evaluation of OPE via SharpeRatio@k, we have also integrated this metric into an open-source software, SCOPE-RL (https://github.com/hakuhodo-technologies/scope-rl). Employing SharpeRatio@k and SCOPE-RL, we conduct comprehensive benchmarking experiments on various estimators and RL tasks, focusing on their risk-return tradeoff. These experiments offer several interesting directions and suggestions for future OPE research.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

35 of 45 references.

References · 35

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  10. 2019Batch Policy Learning under Constraintscited by 244
  11. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  12. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
23 more
  1. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  2. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  3. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  4. 2020Batch Value-function Approximation with Only Realizabilitycited by 104
  5. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  6. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  7. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  8. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  9. 2020Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluationcited by 77
  10. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  11. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  12. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 58
  13. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 54
  14. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  15. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  16. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  17. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  18. 2022Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Modelcited by 30
  19. 2019Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learningcited by 28
  20. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  21. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  22. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  23. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstractioncited by 17
  2. 2024POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decompositioncited by 13
  3. 2024Long-term Off-Policy Evaluation and Learningcited by 9
  4. 2024Effective Off-Policy Evaluation and Learning in Contextual Combinatorial Banditscited by 8
  5. 2024Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with Itcited by 5
  6. 2025Off-Policy Evaluation and Learning for Matching Marketscited by 2
  7. 2024Balancing Immediate Revenue and Future Off-Policy Evaluation in Coupon Allocationcited by 1
  8. 2025Prompt Optimization with Logged Bandit Datacited by 1
  9. 2026Off-Policy Learning with Limited Supplycited by 1
  10. 2025DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effectscited by 0
  11. 2025Evaluation-Aware Reinforcement Learningcited by 0
  12. 2026Cross-Domain Off-Policy Evaluation and Learning for Contextual Banditscited by 0
1 more
  1. 2026Learning from the Test: Self-Referential Differential Testing for Deep RL Agentscited by 0