# Exploratory Optimal Stopping: A Singular Control Formulation

arXiv: https://arxiv.org/abs/2408.09335 · HTML: https://mlsanity.com/paper/2408.09335/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 30 of its 73 references that are in the map.
- First posted on arXiv 18 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 30)

- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Making Deep Q-learning methods robust to time discretization](https://mlsanity.com/paper/1901.09732/) (2019)
- [Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon](https://mlsanity.com/paper/2011.10300/) (2020)
- [Dealing with Sparse Rewards in Reinforcement Learning](https://mlsanity.com/paper/1910.09281/) (2019)
- [Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon](https://mlsanity.com/paper/2006.15316/) (2020)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls](https://mlsanity.com/paper/2104.09311/) (2021)
- [Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models](https://mlsanity.com/paper/2112.10264/) (2021)
- [Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.02409/) (2022)
- [Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equations](https://mlsanity.com/paper/2406.00612/) (2024)
- [Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching](https://mlsanity.com/paper/2404.17939/) (2024)
- [Linear convergence of a policy gradient method for some finite horizon continuous time control problems](https://mlsanity.com/paper/2203.11758/) (2022)
- [Randomized optimal stopping algorithms and their convergence analysis](https://mlsanity.com/paper/2002.00816/) (2020)
- [Learning to reflect: A unifying approach for data-driven stochastic control strategies](https://mlsanity.com/paper/2104.11496/) (2021)
- [Data-driven rules for multidimensional reflection problems](https://mlsanity.com/paper/2311.06639/) (2023)
- [Reinforcement Learning for optimal dividend problem under diffusion model](https://mlsanity.com/paper/2309.10242/) (2023)
- [Data-driven optimal stopping: A pure exploration analysis](https://mlsanity.com/paper/2312.05880/) (2023)
- [Singular Control of (Reflected) Brownian Motion: A Computational Method Suitable for Queueing Applications](https://mlsanity.com/paper/2312.11823/) (2023)
- [A solvable two-dimensional singular stochastic control problem with non convex costs](https://mlsanity.com/paper/1411.2428/) (2014)
- [Irreversible Investment under L'evy Uncertainty: an Equation for the Optimal Boundary](https://mlsanity.com/paper/1411.2395/) (2014)
- [Multidimensional singular control and related Skorokhod problem: sufficient conditions for the characterization of optimal controls](https://mlsanity.com/paper/2103.08487/) (2021)
- [An Offline Learning Approach to Propagator Models](https://mlsanity.com/paper/2309.02994/) (2023)
- [Subgame-Perfect Equilibria in Stochastic Timing Games](https://mlsanity.com/paper/1409.4597/) (2014)
- [Stochastic nonzero-sum games: a new connection between singular control and optimal stopping](https://mlsanity.com/paper/1601.05709/) (2016)
- [On the value of non-Markovian Dynkin games with partial and asymmetric information](https://mlsanity.com/paper/2007.10643/) (2020)

## Built on by (most cited first, 9 of 9)

- [Continuous-time reinforcement learning for optimal switching over multiple regimes](https://mlsanity.com/paper/2512.04697/) (2025)
- [Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems](https://mlsanity.com/paper/2606.11798/) (2026)
- [A Reinforcement Learning Framework for Some Singular Stochastic Control Problems](https://mlsanity.com/paper/2506.22203/) (2025)
- [Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergence](https://mlsanity.com/paper/2506.08121/) (2025)
- [Reinforcement learning for irreversible reinsurance problems: the randomized singular control approach](https://mlsanity.com/paper/2512.02769/) (2025)
- [Robust Exploratory Stopping under Ambiguity in Reinforcement Learning](https://mlsanity.com/paper/2510.10260/) (2025)
- [Reinforcement Learning in Real Option Models](https://mlsanity.com/paper/2602.15643/) (2026)
- [Randomized Optimal Switching Problem and Related Mirror Descent Flow](https://mlsanity.com/paper/2606.12875/) (2026)
- [Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence](https://mlsanity.com/paper/2609.14972/) (2026)

## Abstract (arXiv)

This paper explores continuous-time and state-space optimal stopping problems from a reinforcement learning perspective. We begin by formulating the stopping problem using randomized stopping times, where the decision maker's control is represented by the probability of stopping within a given time-specifically, a bounded, non-decreasing, c`adl`ag control process. To encourage exploration and facilitate learning, we introduce a regularized version of the problem by penalizing the performance criterion with the cumulative residual entropy of the randomized stopping time. The regularized problem takes the form of an (n+1)-dimensional degenerate singular stochastic control with finite-fuel, where the regularized free boundary becomes the graph of a function mapping the state variable of the original stopping problem into the probability of stopping. We address this singular control problem through the dynamic programming principle, which enables us to identify the unique optimal exploratory strategy. Finally, we propose both model-based and model-free reinforcement learning algorithms tailored for exploratory optimal stopping problems. We establish policy improvement guarantees for the proposed algorithms. Moreover, the model-free method is of actor-critic type and it is scalable in high-dimensions under neural network parameterization.