mlsanityData to 7 Oct 2026

Continuous-time reinforcement learning: ellipticity enables model-free value function approximation

arXiv 2602.06930 ↗First posted 6 Feb 2026cs.LG math.OC math.STcited by 0

Not on the map. Why?

Abstract (arXiv)

We study off-policy reinforcement learning for controlling continuous-time Markov diffusion processes with discrete-time observations and actions. We consider model-free algorithms with function approximation that learn value and advantage functions directly from data, without unrealistic structural assumptions on the dynamics. Leveraging the ellipticity of the diffusions, we establish a new class of Hilbert-space positive definiteness and boundedness properties for the Bellman operators. Based on these properties, we propose the Sobolev-prox fitted q-learning algorithm, which learns value and advantage functions by iteratively solving least-squares regression problems. We derive oracle inequalities for the estimation error, governed by (i) the best approximation error of the function classes, (ii) their localized complexity, (iii) exponentially decaying optimization error, and (iv) numerical discretization error. These results identify ellipticity as a key structural property that renders reinforcement learning with function approximation for Markov diffusions no harder than supervised learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 54 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

29 of 54 references.

References · 29

  1. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
  2. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  3. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  4. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 129
  5. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  6. 2020Batch Value-function Approximation with Only Realizabilitycited by 103
  7. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  8. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  9. 2022q-Learning in Continuous Timecited by 68
  10. 2019Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learningcited by 67
  11. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
  12. 2021Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimalitycited by 40
17 more
  1. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  2. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  3. 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 31
  4. 2022Target Network and Truncation Overcome The Deadly Triad in Q-Learningcited by 31
  5. 2023Policy Optimization for Continuous Reinforcement Learningcited by 30
  6. 2021Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networkscited by 27
  7. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 18
  8. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 18
  9. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learningcited by 10
  10. 2024Regret of exploratory policy improvement and q-learningcited by 10
  11. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 9
  12. 2023Data-Driven Merton's Strategies via Policy Randomizationcited by 6
  13. 2022Linear convergence of a policy gradient method for some finite horizon continuous time control problemscited by 4
  14. 2024On Bellman equations for continuous-time policy evaluation I: discretization and approximationcited by 4
  15. 2025Statistical guarantees for continuous-time policy evaluation: blessing of ellipticity and new tradeoffscited by 3
  16. 2025Is RL fine-tuning harder than regression? A PDE learning approach for diffusion modelscited by 3
  17. 2024Robust Reinforcement Learning under Diffusion Models for Data with Jumpscited by 2

Built on by

Not cited yet.