mlsanityData to 7 Oct 2026

Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach

arXiv 2108.06655 ↗First posted 15 Aug 2021cs.LG q-fin.MFcited by 62

Not on the map. Why?

Abstract (arXiv)

We propose a unified framework to study policy evaluation (PE) and the associated temporal difference (TD) methods for reinforcement learning in continuous time and space. We show that PE is equivalent to maintaining the martingale condition of a process. From this perspective, we find that the mean--square TD error approximates the quadratic variation of the martingale and thus is not a suitable objective for PE. We present two methods to use the martingale characterization for designing PE algorithms. The first one minimizes a "martingale loss function", whose solution is proved to be the best approximation of the true value function in the mean--square sense. This method interprets the classical gradient Monte-Carlo algorithm. The second method is based on a system of equations called the "martingale orthogonality conditions" with test functions. Solving these equations in different ways recovers various classical TD algorithms, such as TD(λ), LSTD, and GTD. Different choices of test functions determine in what sense the resulting solutions approximate the true value function. Moreover, we prove that any convergent time-discretized algorithm converges to its continuous-time counterpart as the mesh size goes to zero, and we provide the convergence rate. We demonstrate the theoretical results and corresponding algorithms with numerical experiments and applications.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 6 of its 49 references that are in the map. First posted on arXiv 15 Aug 2021.

Builds on

6 of 49 references.

References · 6

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Solving high-dimensional partial differential equations using deep learningcited by 549
  3. 2018Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equationscited by 233
  4. 2020Entropy Regularization for Mean Field Games with Learningcited by 50
  5. 2020State-Dependent Temperature Control for Langevin Diffusionscited by 17
  6. 2019Deep backward schemes for high-dimensional nonlinear PDEscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 6; 2024: 17; 2025: 17; 2026: 16.

62 papers.

Sort:

By area:

  1. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  2. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  3. 2022q-Learning in Continuous Timecited by 68
  4. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  5. 2023Policy Optimization for Continuous Reinforcement Learningcited by 30
  6. 2023Actor-Critic learning for mean-field control in continuous timecited by 19
  7. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learningcited by 19
  8. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 18
  9. 2023Continuous-time q-learning for mean-field control problemscited by 17
  10. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 15
  11. 2025Provable Maximum Entropy Manifold Exploration via Diffusion Modelscited by 12
  12. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 10
38 more
  1. 2024Regret of exploratory policy improvement and q-learningcited by 10
  2. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 9
  3. 2022Logarithmic regret bounds for continuous-time average-reward Markov decision processescited by 7
  4. 2024Control randomisation approach for policy gradient and application to reinforcement learning in optimal switchingcited by 7
  5. 2024Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropycited by 7
  6. 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 7
  7. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and Statecited by 7
  8. 2023Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spacescited by 6
  9. 2023Data-Driven Merton's Strategies via Policy Randomizationcited by 6
  10. 2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 6
  11. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 5
  12. 2024Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learningcited by 5
  13. 2024Exploratory Optimal Stopping: A Singular Control Formulationcited by 5
  14. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundationscited by 5
  15. 2024On Bellman equations for continuous-time policy evaluation I: discretization and approximationcited by 4
  16. 2025Continuous-time reinforcement learning for optimal switching over multiple regimescited by 4
  17. 2022Square-root regret bounds for continuous-time episodic Markov decision processescited by 3
  18. 2024Reinforcement Learning for Intensity Control: An Application to Choice-Based Network Revenue Managementcited by 3
  19. 2025A Temporal Difference Method for Stochastic Continuous Dynamicscited by 3
  20. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 3
  21. 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedulecited by 3
  22. 2026Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithmscited by 3
  23. 2026Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problemscited by 3
  24. 2024Reinforcement Learning for Jump-Diffusions, with Financial Applicationscited by 2
  25. 2024On the grid-sampling limit SDEcited by 2
  26. 2024Robust Reinforcement Learning under Diffusion Models for Data with Jumpscited by 2
  27. 2025Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problemscited by 2
  28. 2025Continuous-Time Value Iteration for Multi-Agent Reinforcement Learningcited by 2
  29. 2025Continuous-Time Reinforcement Learning for Asset-Liability Managementcited by 2
  30. 2022Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-offcited by 1
  31. 2023Discrete-Time Mean-Variance Strategy Based on Reinforcement Learningcited by 1
  32. 2024A random measure approach to reinforcement learning in continuous timecited by 1
  33. 2024Multi-Task Dynamic Pricing in Credit Market with Contextual Informationcited by 1
  34. 2025Exploratory Utility Maximization Problem with Tsallis Entropycited by 1
  35. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  36. 2025A Robust Model-Based Approach for Continuous-Time Policy Evaluation with Unknown L'evy Process Dynamicscited by 0
  37. 2025Periodic evaluation of defined-contribution pension fund: A dynamic risk measure approachcited by 0
  38. 2025Robust Exploratory Stopping under Ambiguity in Reinforcement Learningcited by 0

50 most cited of 62 All 62 (JSON)