mlsanityData to 14 Oct 2026

Reinforcement Learning with Random Time Horizons

arXiv 2506.00962 ↗First posted 1 Jun 2025cs.LG math.OC math.PRcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We extend the standard reinforcement learning framework to random time horizons. While the classical setting typically assumes finite and deterministic or infinite runtimes of trajectories, we argue that multiple real-world applications naturally exhibit random (potentially trajectory-dependent) stopping times. Since those stopping times typically depend on the policy, their randomness has an effect on policy gradient formulas, which we (mostly for the first time) derive rigorously in this work both for stochastic and deterministic policies. We present two complementary perspectives, trajectory or state-space based, and establish connections to optimal control theory. Our numerical experiments demonstrate that using the proposed formulas can significantly improve optimization convergence compared to traditional approaches.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 29 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

15 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Finite-Sample Analysis of the Monte Carlo Exploring Starts Algorithm for Reinforcement Learning3 Oct

Other references · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2022An optimal control perspective on diffusion-based generative modelingcited by 147
  6. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  7. 2020Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path spacecited by 82
  8. 2016Unifying task specification in reinforcement learningcited by 41
  9. 2019Is the Policy Gradient a Gradient?cited by 40
  10. 2021Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networkscited by 29
  11. 2020Steady State Analysis of Episodic Reinforcement Learningcited by 21
  12. 2022Connecting Stochastic Optimal Control and Reinforcement Learningcited by 5
2 more
  1. 2022Improving control based importance sampling strategies for metastable diffusions via adapted metadynamicscited by 4
  2. 2023Online Reinforcement Learning with Uncertain Episode Lengthscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Why Policy Gradient Algorithms Work for Undiscounted Total-Reward MDPscited by 4
  2. 2026Rare Event Analysis via Stochastic Optimal Controlcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control
  2. 2026Learning from the Near Future: Temporal Self-Distillation for RLVR
  3. 2026Mathematical methods of reinforcement learning
  4. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
  5. 2024Control in Stochastic Environment with Delays: A Model-based Reinforcement Learning Approach
  6. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  7. 2024Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching
  8. 2026Train Short, Act Long: Multistability and the Horizon Independence of Recurrent Policies