mlsanityData to 14 Oct 2026

Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data

arXiv 2306.14063 ↗First posted 24 Jun 2023cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Developing theoretical guarantees on the sample complexity of offline RL methods is an important step towards making data-hungry RL algorithms practically viable. Currently, most results hinge on unrealistic assumptions about the data distribution -- namely that it comprises a set of i.i.d. trajectories collected by a single logging policy. We consider a more general setting where the dataset may have been gathered adaptively. We develop theory for the TMIS Offline Policy Evaluation (OPE) estimator in this generalized setting for tabular MDPs, deriving high-probability, instance-dependent bounds on its estimation error. We also recover minimax-optimal offline learning in the adaptive setting. Finally, we conduct simulations to empirically analyze the behavior of these estimators under adaptive and non-adaptive regimes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 34 references that are in the map. First posted on arXiv 24 Jun 2023.

Builds on

29 of 34 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Offline Reinforcement Learning with Differential Privacy2 Jun
  2. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  3. 2022The Challenges of Exploration for Offline Reinforcement Learning27 Jan
  4. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  5. 2020Batch Value-function Approximation with Only Realizability11 Aug
  6. 2020On Reward-Free Reinforcement Learning with Linear Function Approximation19 Jun

Other references · 23

  1. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  2. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  4. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  5. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  6. 2019Batch Policy Learning under Constraintscited by 244
  7. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  8. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 176
  9. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  10. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  11. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  12. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 91
11 more
  1. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  2. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  3. 2022Sample-Efficient Reinforcement Learning with loglog(T) Switching Costcited by 32
  4. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  5. 2020Optimal Off-Policy Evaluation from Multiple Logging Policiescited by 30
  6. 2019Are sample means in multi-armed bandits positively or negatively biased?cited by 24
  7. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  8. 2018Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matterscited by 13
  9. 2022Near-Optimal Deployment Efficiency in Reward-Free Reinforcement Learning with Linear Function Approximationcited by 13
  10. 2022Off-policy estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiencycited by 10
  11. 2021Assessing the Performance of Online Students -- New Data, New Approaches, Improved Accuracycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  2. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
  3. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  4. 2022Offline RL Policies Should be Trained to be Adaptive
  5. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
  6. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  7. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  8. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning