# Robust Exploratory Stopping under Ambiguity in Reinforcement Learning

arXiv: https://arxiv.org/abs/2510.10260 · HTML: https://mlsanity.com/paper/2510.10260/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 75 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 25 of 25)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [Robust Reinforcement Learning using Offline Data](https://mlsanity.com/paper/2208.05129/) (2022)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model](https://mlsanity.com/paper/1610.03518/) (2016)
- [Deep splitting method for parabolic PDEs](https://mlsanity.com/paper/1907.03452/) (2019)
- [Reinforcement Learning under Model Mismatch](https://mlsanity.com/paper/1706.04711/) (2017)
- [Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage](https://mlsanity.com/paper/2305.09659/) (2023)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Solving high-dimensional optimal stopping problems using deep learning](https://mlsanity.com/paper/1908.01602/) (2019)
- [Approximation error analysis of some deep backward schemes for nonlinear PDEs](https://mlsanity.com/paper/2006.01496/) (2020)
- [Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.02409/) (2022)
- [Full error analysis of the random deep splitting method for nonlinear parabolic PDEs and PIDEs](https://mlsanity.com/paper/2405.05192/) (2024)
- [Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems](https://mlsanity.com/paper/2209.07059/) (2022)
- [Learning to Optimally Stop Diffusion Processes, with Financial Applications](https://mlsanity.com/paper/2408.09242/) (2024)
- [Exploratory Optimal Stopping: A Singular Control Formulation](https://mlsanity.com/paper/2408.09335/) (2024)
- [Sensitivity of robust optimization problems under drift and volatility uncertainty](https://mlsanity.com/paper/2311.11248/) (2023)
- [Numerical method for nonlinear Kolmogorov PDEs via sensitivity analysis](https://mlsanity.com/paper/2403.11910/) (2024)
- [Robust dividend policy: Equivalence of Epstein-Zin and Maenhout preferences](https://mlsanity.com/paper/2406.12305/) (2024)

## Abstract (arXiv)

We propose and analyze a continuous-time robust reinforcement learning framework for optimal stopping under ambiguity. In this framework, an agent chooses a robust exploratory stopping time motivated by two objectives: robust decision-making under ambiguity and learning about the unknown environment. Here, ambiguity refers to considering multiple probability measures dominated by a reference measure, reflecting the agent's awareness that the reference measure representing her learned belief about the environment would be erroneous. Using the g-expectation framework, we reformulate the optimal stopping problem under ambiguity as a robust exploratory control problem with Bernoulli distributed controls. We then characterize the optimal Bernoulli distributed control via backward stochastic differential equations and, based on this, construct the robust exploratory stopping time that approximates the optimal stopping time under ambiguity. Last, we establish a policy iteration theorem and implement it as a reinforcement learning algorithm. Numerical experiments demonstrate the convergence, robustness, and scalability of our reinforcement learning algorithm across different levels of ambiguity and exploration.