mlsanityData to 14 Oct 2026

On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures

arXiv 2501.02089 ↗First posted 3 Jan 2025cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

This article reviews the recent advances on the statistical foundation of reinforcement learning (RL) in the offline and low-adaptive settings. We will start by arguing why offline RL is the appropriate model for almost any real-life ML problems, even if they have nothing to do with the recent AI breakthroughs that use RL. Then we will zoom into two fundamental problems of offline RL: offline policy evaluation (OPE) and offline policy learning (OPL). It may be surprising to people that tight bounds for these problems were not known even for tabular and linear cases until recently. We delineate the differences between worst-case minimax bounds and instance-dependent bounds. We also cover key algorithmic ideas and proof techniques behind near-optimal instance-dependent methods in OPE and OPL. Finally, we discuss the limitations of offline RL and review a burgeoning problem of low-adaptive exploration which addresses these limitations by providing a sweet middle ground between offline and online RL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 7 of its 21 references that are in the map. First posted on arXiv 3 Jan 2025.

Builds on

7 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation23 Nov
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar

Other references · 5

  1. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  2. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 100
  3. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  4. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  5. 2020Q-Learning in Regularized Mean-field Gamescited by 42

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theorycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  2. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  3. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis
  4. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
  5. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  6. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  7. 2020MOPO: Model-based Offline Policy Optimization
  8. 2020MOReL : Model-Based Offline Reinforcement Learning