mlsanityData to 7 Oct 2026

SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluation

arXiv 2311.18206 ↗First posted 30 Nov 2023cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

This paper introduces SCOPE-RL, a comprehensive open-source Python software designed for offline reinforcement learning (offline RL), off-policy evaluation (OPE), and selection (OPS). Unlike most existing libraries that focus solely on either policy learning or evaluation, SCOPE-RL seamlessly integrates these two key aspects, facilitating flexible and complete implementations of both offline RL and OPE processes. SCOPE-RL put particular emphasis on its OPE modules, offering a range of OPE estimators and robust evaluation-of-OPE protocols. This approach enables more in-depth and reliable OPE compared to other packages. For instance, SCOPE-RL enhances OPE by estimating the entire reward distribution under a policy rather than its mere point-wise expected value. Additionally, SCOPE-RL provides a more thorough evaluation-of-OPE by presenting the risk-return tradeoff in OPE results, extending beyond mere accuracy evaluations in existing OPE literature. SCOPE-RL is designed with user accessibility in mind. Its user-friendly APIs, comprehensive documentation, and a variety of easy-to-follow examples assist researchers and practitioners in efficiently implementing and experimenting with various offline RL methods and OPE estimators, tailored to their specific problem contexts. The documentation of SCOPE-RL is available at https://scope-rl.readthedocs.io/en/latest/.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 60 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

44 of 60 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning3 Nov
  2. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  3. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  4. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
  5. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  6. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  7. 2020Batch Value-function Approximation with Only Realizability11 Aug
  8. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 35

  1. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  2. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  3. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  4. 2019Batch Policy Learning under Constraintscited by 242
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  7. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  8. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  9. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  10. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  11. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  12. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
23 more
  1. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  2. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  3. 2020Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluationcited by 77
  4. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  5. 2018RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertisingcited by 65
  6. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 57
  7. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
  8. 2018Horizon: Facebook's Open Source Applied Reinforcement Learning Platformcited by 48
  9. 2021Universal Off-Policy Evaluationcited by 43
  10. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  11. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  12. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  13. 2022Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Modelcited by 30
  14. 2021Off-Policy Risk Assessment in Contextual Banditscited by 29
  15. 2019Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learningcited by 28
  16. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  17. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  18. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  19. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  20. 2023Off-Policy Evaluation of Ranking Policies under Diverse User Behaviorcited by 16
  21. 2022Off-Policy Risk Assessment in Markov Decision Processescited by 7
  22. 2021SOPE: Spectrum of Off-Policy Estimatorscited by 5
  23. 2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstractioncited by 17
  2. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocolcited by 6
  3. 2025Prompt Optimization with Logged Bandit Datacited by 1
  4. 2025GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agentscited by 0
  5. 2026When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guidecited by 0
  6. 2026Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PyCFRL: A Python library for counterfactually fair offline reinforcement learning via sequential data preprocessing
  2. 2025A Clean Slate for Offline Reinforcement Learning
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  4. 2026Active Offline-to-Online Reinforcement Learning
  5. 2025PyTupli: A Scalable Infrastructure for Collaborative Offline Reinforcement Learning Projects
  6. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  7. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  8. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning