# A Robust Model-Based Approach for Continuous-Time Policy Evaluation with Unknown L'evy Process Dynamics

arXiv: https://arxiv.org/abs/2504.01482 · HTML: https://mlsanity.com/paper/2504.01482/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 30 of its 64 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 30)

- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Neural Jump Stochastic Differential Equations](https://mlsanity.com/paper/1905.10403/) (2019)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Probabilistic Forecasting with Stochastic Interpolants and F"ollmer Processes](https://mlsanity.com/paper/2403.13724/) (2024)
- [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020)
- [Solving Inverse Stochastic Problems from Discrete Particle Observations Using the Fokker-Planck Equation and Physics-informed Neural Networks](https://mlsanity.com/paper/2008.10653/) (2020)
- [On the model-based stochastic value gradient for continuous reinforcement learning](https://mlsanity.com/paper/2008.12775/) (2020)
- [Learning effective stochastic differential equations from microscopic simulations: linking stochastic numerics to deep learning](https://mlsanity.com/paper/2106.09004/) (2021)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Learning Stochastic Dynamical System via Flow Map Operator](https://mlsanity.com/paper/2305.03874/) (2023)
- [A Training-Free Conditional Diffusion Model for Learning Stochastic Dynamical Systems](https://mlsanity.com/paper/2410.03108/) (2024)
- [Learning Stochastic Dynamics with Statistics-Informed Neural Network](https://mlsanity.com/paper/2202.12278/) (2022)
- [Reward Centering](https://mlsanity.com/paper/2405.09999/) (2024)
- [One-Shot Learning of Stochastic Differential Equations with Data Adapted Kernels](https://mlsanity.com/paper/2209.12086/) (2022)
- [Neural network stochastic differential equation models with applications to financial data forecasting](https://mlsanity.com/paper/2111.13164/) (2021)
- [Dynamic Decision Frequency with Continuous Options](https://mlsanity.com/paper/2212.04407/) (2022)
- [Learning Stochastic Behaviour from Aggregate Data](https://mlsanity.com/paper/2002.03513/) (2020)
- [Modeling Unknown Stochastic Dynamical System via Autoencoder](https://mlsanity.com/paper/2312.10001/) (2023)
- [Stationary Density Estimation of Ito Diffusions Using Deep Learning](https://mlsanity.com/paper/2109.03992/) (2021)
- [A mean field game inverse problem](https://mlsanity.com/paper/2007.11551/) (2020)
- [Extracting stochastic dynamical systems with α-stable L'evy noise from data](https://mlsanity.com/paper/2109.14881/) (2021)
- [DynGMA: a robust approach for learning stochastic differential equations from data](https://mlsanity.com/paper/2402.14475/) (2024)
- [Designing Rewards for Fast Learning](https://mlsanity.com/paper/2205.15400/) (2022)
- [PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation](https://mlsanity.com/paper/2405.12535/) (2024)
- [An efficient Wasserstein-distance approach for reconstructing jump-diffusion processes using parameterized neural networks](https://mlsanity.com/paper/2406.01653/) (2024)
- [A Policy Iteration Method for Inverse Mean Field Games](https://mlsanity.com/paper/2409.06184/) (2024)
- [Weak Collocation Regression for Inferring Stochastic Dynamics with L'evy Noise](https://mlsanity.com/paper/2403.08292/) (2024)

## Abstract (arXiv)

This paper develops a model-based framework for continuous-time policy evaluation (CTPE) in reinforcement learning, incorporating both Brownian and L'evy noise to model stochastic dynamics influenced by rare and extreme events. Our approach formulates the policy evaluation problem as solving a partial integro-differential equation (PIDE) for the value function with unknown coefficients. A key challenge in this setting is accurately recovering the unknown coefficients in the stochastic dynamics, particularly when driven by L'evy processes with heavy tail effects. To address this, we propose a robust numerical approach that effectively handles both unbiased and censored trajectory datasets. This method combines maximum likelihood estimation with an iterative tail correction mechanism, improving the stability and accuracy of coefficient recovery. Additionally, we establish a theoretical bound for the policy evaluation error based on coefficient recovery error. Through numerical experiments, including a real-data BTC price experiment, we demonstrate the effectiveness and robustness of our method in recovering heavy-tailed L'evy dynamics and verify the theoretical error analysis in policy evaluation.