# Zap Q-Learning for Optimal Stopping Time Problems

arXiv: https://arxiv.org/abs/1904.11538 · HTML: https://mlsanity.com/paper/1904.11538/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2019.

## Built on by (most cited first, 3 of 3)

- [Deep Reinforcement Learning for Optimal Stopping with Application in Financial Engineering](https://mlsanity.com/paper/2105.08877/) (2021)
- [Optimal Stopping with Gaussian Processes](https://mlsanity.com/paper/2209.14738/) (2022)
- [Deep Optimal Timing Strategies for Time Series](https://mlsanity.com/paper/2310.05479/) (2023)

## Abstract (arXiv)

The objective in this paper is to obtain fast converging reinforcement learning algorithms to approximate solutions to the problem of discounted cost optimal stopping in an irreducible, uniformly ergodic Markov chain, evolving on a compact subset of ℝn. We build on the dynamic programming approach taken by Tsitsikilis and Van Roy, wherein they propose a Q-learning algorithm to estimate the optimal state-action value function, which then defines an optimal stopping rule. We provide insights as to why the convergence rate of this algorithm can be slow, and propose a fast-converging alternative, the "Zap-Q-learning" algorithm, designed to achieve optimal rate of convergence. For the first time, we prove the convergence of the Zap-Q-learning algorithm under the assumption of linear function approximation setting. We use ODE analysis for the proof, and the optimal asymptotic variance property of the algorithm is reflected via fast convergence in a finance example.