mlsanityData to 14 Oct 2026

Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation

arXiv 2604.18972 ↗First posted 21 Apr 2026stat.ML cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

We study finite-horizon continuous-time policy evaluation from discrete closed-loop trajectories under time-inhomogeneous dynamics. The target value surface solves a backward parabolic equation, but the Bellman baseline obtained from one-step recursion is only first-order in the grid width. We estimate the time-dependent generator from multi-step transitions using moment-matching coefficients that cancel lower-order truncation terms, and combine the resulting surrogate with backward regression. The main theory gives an end-to-end decomposition into generator misspecification, projection error, pooling bias, finite-sample error, and start-up error, together with a decision-frequency regime map explaining when higher-order gains should be visible. Across calibration studies, four-scale benchmarks, feature and start-up ablations, and gain-mismatch stress tests, the second-order estimator consistently improves on the Bellman baseline and remains stable in the regime where the theory predicts visible gains. These results position high-order generator regression as an interpretable continuous-time policy-evaluation method with a clear operating region.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 44 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

18 of 44 references.

References · 18

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  4. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  6. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  7. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  8. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  9. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  10. 2022q-Learning in Continuous Timecited by 91
  11. 2019Data-driven approximation of the Koopman generator: Model reduction, system identification, and controlcited by 85
  12. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
6 more
  1. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  2. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  3. 2020Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approachcited by 35
  4. 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 31
  5. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  6. 2020Reliable Off-policy Evaluation for Reinforcement Learningcited by 9

Built on by

Not cited yet.