mlsanityData to 7 Oct 2026

Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation

arXiv 2008.07146 ↗First posted 17 Aug 2020cs.LG stat.MLcited by 77

Not on the map. Why?

Abstract (arXiv)

Off-policy evaluation (OPE) aims to estimate the performance of hypothetical policies using data generated by a different policy. Because of its huge potential impact in practice, there has been growing research interest in this field. There is, however, no real-world public dataset that enables the evaluation of OPE, making its experimental studies unrealistic and irreproducible. With the goal of enabling realistic and reproducible OPE research, we present Open Bandit Dataset, a public logged bandit dataset collected on a large-scale fashion e-commerce platform, ZOZOTOWN. Our dataset is unique in that it contains a set of multiple logged bandit datasets collected by running different policies on the same platform. This enables experimental comparisons of different OPE estimators for the first time. We also develop Python software called Open Bandit Pipeline to streamline and standardize the implementation of batch bandit algorithms and OPE. Our open data and software will contribute to fair and transparent OPE research and help the community identify fruitful research directions. We provide extensive benchmark experiments of existing OPE estimators using our dataset and software. The results open up essential challenges and new avenues for future OPE research.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 16; 2024: 11; 2025: 15; 2026: 23.

77 papers.

Sort:

By area:

  1. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
  2. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  3. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  4. 2022Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Modelcited by 30
  5. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  6. 2023Acquisition Conditioned Oracle for Nongreedy Active Feature Acquisitioncited by 17
  7. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  8. 2023Off-Policy Evaluation of Ranking Policies under Diverse User Behaviorcited by 16
  9. 2023On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n Recommendationcited by 13
  10. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  11. 2024POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decompositioncited by 13
  12. 2024Optimal Baseline Corrections for Off-Policy Contextual Banditscited by 13
38 more
  1. 2022Scaling Gaussian Process Optimization by Evaluating a Few Unique Candidates Multiple Timescited by 12
  2. 2023Safe Deployment for Counterfactual Learning to Rank with Exposure-Based Risk Minimizationcited by 11
  3. 2023Ad-load Balancing via Off-policy Learning in a Content Marketplacecited by 11
  4. 2023Practical Bandits: An Industry Perspectivecited by 9
  5. 2023Marginal Density Ratio for Off-Policy Evaluation in Contextual Banditscited by 9
  6. 2024Long-term Off-Policy Evaluation and Learningcited by 9
  7. 2024Delta-rm OPE: Off-Policy Estimation with Pairs of Policiescited by 9
  8. 2024EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systemscited by 8
  9. 2024Effective Off-Policy Evaluation and Learning in Contextual Combinatorial Banditscited by 8
  10. 2023Learning Action Embeddings for Off-Policy Evaluationcited by 7
  11. 2022Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actionscited by 6
  12. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  13. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 5
  14. 2022Synthetic Data-Based Simulators for Recommender Systems: A Surveycited by 5
  15. 2024Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with Itcited by 5
  16. 2025Neural Logistic Banditscited by 5
  17. 2025Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformerscited by 4
  18. 2023When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspectivecited by 3
  19. 2026CIVeX: Causal Intervention Verification for Language Agentscited by 3
  20. 2026Counterfactual Tool Ranking under Utility, Cost, and Privilege Constraintscited by 3
  21. 2022Extending Open Bandit Pipeline to Simulate Industry Challengescited by 2
  22. 2023Distributional Off-Policy Evaluation for Slate Recommendationscited by 2
  23. 2023CATE Lasso: Conditional Average Treatment Effect Estimation with High-Dimensional Linear Regressioncited by 2
  24. 2024Automated Off-Policy Estimator Selection via Supervised Learningcited by 2
  25. 2025Clustering Context in Off-Policy Evaluationcited by 2
  26. 2026Decision Support for Marketplace Policies under Incomplete Evidence: From Replay to Launch Readinesscited by 2
  27. 2023A Field Test of Bandit Algorithms for Recommendations: Understanding the Validity of Assumptions on Human Preferences in Multi-armed Banditscited by 1
  28. 2023Doubly Robust Estimator for Off-Policy Evaluation with Large Action Spacescited by 1
  29. 2025Prompt Optimization with Logged Bandit Datacited by 1
  30. 2025Managing Self-Learning Experts under Per-Round Budget Constraintscited by 1
  31. 2025Offline A/B Testing of Slate Recommendation Systems with LLMs: Reducing the Dependency on Pre-Collected User Interaction Datacited by 1
  32. 2026BanditLP: Large-Scale Stochastic Optimization for Personalized Recommendationscited by 1
  33. 2026Off-Policy Learning with Limited Supplycited by 1
  34. 2026Catching a Moving Subspace: Low-Rank Bandits Beyond Stationaritycited by 1
  35. 2026Choosing Online Experiment Designs under Interference in Ads, Recommendations, and Member-Experience Systemscited by 1
  36. 2021Off-Policy Evaluation Using Information Borrowing and Context-Based Switchingcited by 0
  37. 2022Situating Recommender Systems in Practice: Towards Inductive Learning and Incremental Updatescited by 0
  38. 2024AExGym: Benchmarks and Environments for Adaptive Experimentationcited by 0

50 most cited of 77 All 77 (JSON)