# Reinforcement Learning Design for Quickest Change Detection

arXiv: https://arxiv.org/abs/2403.14109 · HTML: https://mlsanity.com/paper/2403.14109/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 28 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 8 of 8)

- [Approximate information state for approximate planning and reinforcement learning in partially observed systems](https://mlsanity.com/paper/2010.08843/) (2020)
- [The ODE Method for Asymptotic Statistics in Stochastic Approximation and Reinforcement Learning](https://mlsanity.com/paper/2110.14427/) (2021)
- [Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Approach](https://mlsanity.com/paper/1809.05258/) (2018)
- [Zap Q-Learning With Nonlinear Function Approximation](https://mlsanity.com/paper/1910.05405/) (2019)
- [Stability of Q-Learning Through Design and Optimism](https://mlsanity.com/paper/2307.02632/) (2023)
- [Quickest Change Detection with Non-Stationary Post-Change Observations](https://mlsanity.com/paper/2110.01581/) (2021)
- [Non-Parametric Quickest Mean Change Detection](https://mlsanity.com/paper/2108.11348/) (2021)
- [Structural Results for Partially Observed Markov Decision Processes](https://mlsanity.com/paper/1512.03873/) (2015)

## Built on by (most cited first, 1 of 1)

- [Reinforcement Learning for Optimal Stopping in POMDPs with Application to Quickest Change Detection](https://mlsanity.com/paper/2512.22347/) (2025)

## Abstract (arXiv)

The field of quickest change detection (QCD) concerns design and analysis of algorithms to estimate in real time the time at which an important event takes place, and identify properties of the post-change behavior. It is shown in this paper that approaches based on reinforcement learning (RL) can be adapted based on any "surrogate information state" that is adapted to the observations. Hence we are left to choose both the surrogate information state process and the algorithm. For the former, it is argued that there are many choices available, based on a rich theory of asymptotic statistics for QCD. Two approaches to RL design are considered: (i) Stochastic gradient descent based on an actor-critic formulation. Theory is largely complete for this approach: the algorithm is unbiased, and will converge to a local minimum. However, it is shown that variance of stochastic gradients can be very large, necessitating the need for commensurately long run times; (ii) Q-learning algorithms based on a version of the projected Bellman equation. It is shown that the algorithm is stable, in the sense of bounded sample paths, and that a solution to the projected Bellman equation exists under mild conditions. Numerical experiments illustrate these findings, and provide a roadmap for algorithm design in more general settings.