mlsanityData to 7 Oct 2026

Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces

arXiv 2401.05233 ↗First posted 10 Jan 2024cs.LG cs.IT cs.SYcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce a novel framework for analyzing reinforcement learning (RL) in continuous state-action spaces, and use it to prove fast rates of convergence in both off-line and on-line settings. Our analysis highlights two key stability properties, relating to how changes in value functions and/or policies affect the Bellman operator and occupation measures. We argue that these properties are satisfied in many continuous state-action Markov decision processes, and demonstrate how they arise naturally when using linear function approximation methods. Our analysis offers fresh perspectives on the roles of pessimism and optimism in off-line and on-line RL, and highlights the connection between off-line RL and transfer learning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

24 of 36 references.

References · 24

  1. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  3. 2018Is Q-learning Provably Efficient?cited by 485
  4. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  5. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  7. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  8. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  9. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  10. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 151
  11. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 129
  12. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
12 more
  1. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  2. 2017Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigationcited by 90
  3. 2017Mostly Exploration-Free Algorithms for Contextual Banditscited by 85
  4. 2020Logarithmic Regret for Reinforcement Learning with Linear Function Approximationcited by 79
  5. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  6. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  7. 2021Fast Rates for the Regret of Offline Reinforcement Learningcited by 26
  8. 2021Optimal policy evaluation using kernel-based temporal difference methodscited by 24
  9. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximationcited by 22
  10. 2022On Gap-dependent Bounds for Offline Reinforcement Learningcited by 13
  11. 2017Fast Rates for Empirical Risk Minimization of Strict Saddle Problemscited by 7
  12. 2022Policy evaluation from a single path: Multi-step methods, mixing and mis-specificationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholdscited by 2
  2. 2024Heavy-Ball Momentum Accelerated Actor-Critic With Function Approximationcited by 0
  3. 2026Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processescited by 0