mlsanityData to 7 Oct 2026

Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees

arXiv 2510.04088 ↗First posted 5 Oct 2025cs.LG cs.AI stat.MLcited by 10

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

This article introduces the theory of offline reinforcement learning in large state spaces, where good policies are learned from historical data without online interactions with the environment. Key concepts introduced include expressivity assumptions on function approximation (e.g., Bellman completeness vs. realizability) and data coverage (e.g., all-policy vs. single-policy coverage). A rich landscape of algorithms and results is described, depending on the assumptions one is willing to make and the sample and computational complexity guarantees one wishes to achieve. We also discuss open questions and connections to adjacent areas.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 74 of its 120 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

74 of 120 references.

Closest ancestors · 27

Same direction, newest first

  1. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol11 Feb
  2. 2024Offline Reinforcement Learning: Role of State Aggregation and Trajectory Data25 Mar
  3. 2023Importance-Weighted Offline Learning Done Right27 Sep
  4. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  5. 2023Offline Learning in Markov Games with General Function Approximation6 Feb
  6. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders1 Feb
  7. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning30 Jan
  8. 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
  9. 2022The Role of Coverage in Online Reinforcement Learning9 Oct
  10. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonus1 Jun
  11. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps25 Mar
  12. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
15 more
  1. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  2. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  3. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes28 Oct
  4. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
  5. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  8. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  9. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  10. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  11. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  12. 2020Batch Value-function Approximation with Only Realizability11 Aug
  13. 2020MOPO: Model-based Offline Policy Optimization27 May
  14. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  15. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning11 Feb

Other references · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019Introduction to Online Convex Optimizationcited by 793
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  9. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  10. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  11. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  12. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
35 more
  1. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  2. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  3. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  4. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  5. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  6. 2021Reinforcement learning based recommender systems: A surveycited by 187
  7. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  8. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  9. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  10. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  11. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  12. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  13. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  14. 2021Reward is enough for convex MDPscited by 82
  15. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  16. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  17. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  18. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  19. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  20. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  21. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  22. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  23. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  24. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  25. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  26. 2020A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Settingcited by 24
  27. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  28. 2019Lecture Notes: Selected topics on robust statistical learning theorycited by 17
  29. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  30. 2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 17
  31. 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
  32. 2024On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluationcited by 9
  33. 2022A Complete Characterization of Linear Estimators for Offline Policy Evaluationcited by 7
  34. 2024A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learningcited by 7
  35. 2019Sample-Optimal Parametric Q-Learning Using Linearly Additive Featurescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  2. 2025Outcome-Aware Spectral Feature Learning for Instrumental Variable Regressioncited by 2
  3. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
  4. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 1
  5. 2026A Unifying View of Coverage in Linear Off-Policy Evaluationcited by 1
  6. 2026A Covering Framework for Offline POMDPs Learning using Belief Space Metriccited by 1
  7. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policiescited by 0
  8. 2026Residuals-based Offline Reinforcement Learningcited by 0
  9. 2026Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matchingcited by 0
  10. 2026Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  2. 2023Offline Reinforcement Learning with Additional Covering Distributions
  3. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  4. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  5. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  6. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
  7. 2024Tractable Offline Learning of Regular Decision Processes
  8. 2026On the Complexity of Offline Reinforcement Learning with Q^star-Approximation and Partial Coverage