mlsanityData to 14 Oct 2026

Robust Reinforcement Learning under Diffusion Models for Data with Jumps

arXiv 2411.11697 ↗First posted 18 Nov 2024cs.LG stat.MLcited by 2

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Reinforcement Learning (RL) has proven effective in solving complex decision-making tasks across various domains, but challenges remain in continuous-time settings, particularly when state dynamics are governed by stochastic differential equations (SDEs) with jump components. In this paper, we address this challenge by introducing the Mean-Square Bipower Variation Error (MSBVE) algorithm, which enhances robustness and convergence in scenarios involving significant stochastic noise and jumps. We first revisit the Mean-Square TD Error (MSTDE) algorithm, commonly used in continuous-time RL, and highlight its limitations in handling jumps in state dynamics. The proposed MSBVE algorithm minimizes the mean-square quadratic variation error, offering improved performance over MSTDE in environments characterized by SDEs with jumps. Simulations and formal proofs demonstrate that the MSBVE algorithm reliably estimates the value function in complex settings, surpassing MSTDE's performance when faced with jump processes. These findings underscore the importance of alternative error metrics to improve the resilience and effectiveness of RL algorithms in continuous-time frameworks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 8 of its 22 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

8 of 22 references.

References · 8

  1. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  2. 2022q-Learning in Continuous Timecited by 91
  3. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  4. 2021Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controlscited by 24
  5. 2024Control randomisation approach for policy gradient and application to reinforcement learning in optimal switchingcited by 8
  6. 2024Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropycited by 8
  7. 2023Entropy-Regularized Mean-Variance Portfolio Optimization with Jumpscited by 5
  8. 2023Robust Realized Integrated Beta Estimator with Application to Dynamic Analysis of Integrated Betacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Continuous-time reinforcement learning: ellipticity enables model-free value function approximationcited by 0
  2. 2026An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Time-Constrained Robust MDPs
  2. 2025Adversarial Diffusion for Robust Reinforcement Learning
  3. 2024Model-Free Robust Reinforcement Learning with Sample Complexity Analysis
  4. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  5. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  6. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
  7. 2024Safe and Robust Reinforcement Learning: Principles and Practice
  8. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty