mlsanityData to 7 Oct 2026

Offline Policy Comparison under Limited Historical Agent-Environment Interactions

arXiv 2106.03934 ↗First posted 7 Jun 2021cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

We address the challenge of policy evaluation in real-world applications of reinforcement learning systems where the available historical data is limited due to ethical, practical, or security considerations. This constrained distribution of data samples often leads to biased policy evaluation estimates. To remedy this, we propose that instead of policy evaluation, one should perform policy comparison, i.e. to rank the policies of interest in terms of their value based on available historical data. In addition we present the Limited Data Estimator (LDE) as a simple method for evaluating and comparing policies from a small number of interactions with the environment. According to our theoretical analysis, the LDE is shown to be statistically reliable on policy comparison tasks under mild assumptions on the distribution of the historical data. Additionally, our numerical experiments compare the LDE to other policy evaluation methods on the task of policy ranking and demonstrate its advantage in various settings.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 35 of its 69 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

35 of 69 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  11. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  12. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
23 more
  1. 2019Challenges of Real-World Reinforcement Learningcited by 282
  2. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  3. 2019Batch Policy Learning under Constraintscited by 242
  4. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
  5. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  6. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  7. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  8. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  9. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  10. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  11. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  12. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
  13. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  14. 2015Counterfactual Risk Minimization: Learning from Logged Bandit Feedbackcited by 33
  15. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  16. 2020Steady State Analysis of Episodic Reinforcement Learningcited by 21
  17. 2021Off-policy Confidence Sequencescited by 15
  18. 2020Reliable Off-policy Evaluation for Reinforcement Learningcited by 9
  19. 2020Offline Reinforcement Learning Hands-Oncited by 6
  20. 2017Universal Reinforcement Learning Algorithms: Survey and Experimentscited by 4
  21. 2020Rapidly Personalizing Mobile Health Treatment Policies with Limited Datacited by 4
  22. 2015A Notation for Markov Decision Processescited by 3
  23. 2021pH-RL: A personalization architecture to bring reinforcement learning to health practicecited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 1.

3 papers.

  1. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  2. 2022Examining Policy Entropy of Reinforcement Learning Agents for Personalization Taskscited by 3
  3. 2024Data-Centric Approach to Constrained Machine Learning: A Case Study on Conway's Game of Lifecited by 2