mlsanityData to 13 Oct 2026

Information-Theoretic Considerations in Batch Reinforcement Learning

arXiv 1905.00360 ↗First posted 1 May 2019cs.LG cs.AI stat.MLcited by 287

Not on the map. Why?

Cited by 134 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Value-function approximation methods that operate in batch mode have foundational importance to reinforcement learning (RL). Finite sample guarantees for these methods often crucially rely on two types of assumptions: (1) mild distribution shift, and (2) representation conditions that are stronger than realizability. However, the necessity ("why do we need them?") and the naturalness ("when do they hold?") of such assumptions have largely eluded the literature. In this paper, we revisit these assumptions and provide theoretical results towards answering the above questions, and make steps towards a deeper understanding of value-function approximation.

Key facts. Cited by 287 papers in the map (data through 13 Oct 2026). First posted on arXiv 1 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2019: 1; 2020: 7; 2021: 50; 2022: 55; 2023: 70; 2024: 42; 2025: 37; 2026: 25.

287 papers.

Sort:

By area:

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,782
  2. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 584
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 366
  4. 2020Is Pessimism Provably Efficient for Offline RL?cited by 363
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 277
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 268
  7. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  8. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 161
  10. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 159
  11. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  12. 2021Offline RL Without Off-Policy Evaluationcited by 150
38 more
  1. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  2. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  3. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  4. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  5. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  6. 2022Jump-Start Reinforcement Learningcited by 103
  7. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  8. 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 97
  9. 2022Robust Reinforcement Learning using Offline Datacited by 95
  10. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 94
  11. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 87
  12. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  13. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  14. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  15. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  16. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  17. 2019Provably Efficient Imitation Learning from Observation Alonecited by 78
  18. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  19. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  20. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 75
  21. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 70
  22. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  23. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 64
  24. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
  25. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  26. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  27. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvementcited by 49
  28. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  29. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  30. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  31. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
  32. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 41
  33. 2021An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gapcited by 40
  34. 2022Provable Reinforcement Learning with a Short-Term Memorycited by 40
  35. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  36. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  37. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 35
  38. 2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35

50 most cited of 287 All 287 (JSON)