mlsanityData to 14 Oct 2026

A learning-based approach to stochastic optimal control under reach-avoid constraint

arXiv 2412.16561 ↗First posted 21 Dec 2024math.OC cs.LGcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We develop a model-free approach to optimally control stochastic, Markovian systems subject to a reach-avoid constraint. Specifically, the state trajectory must remain within a safe set while reaching a target set within a finite time horizon. Due to the time-dependent nature of these constraints, we show that, in general, the optimal policy for this constrained stochastic control problem is non-Markovian, which increases the computational complexity. To address this challenge, we apply the state-augmentation technique from arXiv:2402.19360, reformulating the problem as a constrained Markov decision process (CMDP) on an extended state space. This transformation allows us to search for a Markovian policy, avoiding the complexity of non-Markovian policies. To learn the optimal policy without a system model, and using only trajectory data, we develop a log-barrier policy gradient approach. We prove that under suitable assumptions, the policy parameters converge to the optimal parameters, while ensuring that the system trajectories satisfy the stochastic reach-avoid constraint with high probability.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 36 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

19 of 36 references.

References · 19

  1. 2017Constrained Policy Optimizationcited by 983
  2. 2018Reward Constrained Policy Optimizationcited by 417
  3. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  4. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  5. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  6. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  7. 2017Towards Generalization and Simplicity in Continuous Controlcited by 68
  8. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
  9. 2017Sub-sampled Cubic Regularization for Non-convex Optimizationcited by 63
  10. 2021Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learningcited by 49
  11. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  12. 2019Discrete and Continuous Action Representation for Practical RL in Video Gamescited by 39
7 more
  1. 2022Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learningcited by 11
  2. 2023A safe exploration approach to constrained Markov decision processescited by 10
  3. 2023Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methodscited by 9
  4. 2015Receding-horizon Stochastic Model Predictive Control with Hard Input Constraints and Joint State Chance Constraintscited by 8
  5. 2022Data-Driven Chance Constrained Control using Kernel Distribution Embeddingscited by 5
  6. 2023Computing Optimal Joint Chance Constrained Control Policiescited by 3
  7. 2018Worst-case iteration bounds for log barrier methods on problems with nonconvex constraintscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Joint Chance Constrained Safe-Optimal Controlcited by 0
  2. 2026Learning-Based Stochastic Optimal Control with Infinite-Horizon Probabilistic Constraintscited by 0
  3. 2026Provably Safe Sim-to-Real Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time
  2. 2026Stochastic Minimum-Cost Reach-Avoid Reinforcement Learning
  3. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guarantees
  4. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form
  5. 2025Learning Deterministic Policies with Policy Gradients in Constrained Markov Decision Processes
  6. 2025An Optimal Policy for Learning Controllable Dynamics by Exploration
  7. 2025Polynomial-Time Approximability of Constrained Reinforcement Learning
  8. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence