mlsanityData to 14 Oct 2026

Reinforcement Learning Design for Quickest Change Detection

arXiv 2403.14109 ↗First posted 21 Mar 2024math.OC cs.IT math.ITcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The field of quickest change detection (QCD) concerns design and analysis of algorithms to estimate in real time the time at which an important event takes place, and identify properties of the post-change behavior. It is shown in this paper that approaches based on reinforcement learning (RL) can be adapted based on any "surrogate information state" that is adapted to the observations. Hence we are left to choose both the surrogate information state process and the algorithm. For the former, it is argued that there are many choices available, based on a rich theory of asymptotic statistics for QCD. Two approaches to RL design are considered: (i) Stochastic gradient descent based on an actor-critic formulation. Theory is largely complete for this approach: the algorithm is unbiased, and will converge to a local minimum. However, it is shown that variance of stochastic gradients can be very large, necessitating the need for commensurately long run times; (ii) Q-learning algorithms based on a version of the projected Bellman equation. It is shown that the algorithm is stable, in the sense of bounded sample paths, and that a solution to the projected Bellman equation exists under mild conditions. Numerical experiments illustrate these findings, and provide a roadmap for algorithm design in more general settings.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 8 of its 28 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

8 of 28 references.

References · 8

  1. 2020Approximate information state for approximate planning and reinforcement learning in partially observed systemscited by 70
  2. 2021The ODE Method for Asymptotic Statistics in Stochastic Approximation and Reinforcement Learningcited by 32
  3. 2018Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Approachcited by 15
  4. 2019Zap Q-Learning With Nonlinear Function Approximationcited by 13
  5. 2023Stability of Q-Learning Through Design and Optimismcited by 8
  6. 2021Quickest Change Detection with Non-Stationary Post-Change Observationscited by 4
  7. 2021Non-Parametric Quickest Mean Change Detectioncited by 3
  8. 2015Structural Results for Partially Observed Markov Decision Processescited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Reinforcement Learning for Optimal Stopping in POMDPs with Application to Quickest Change Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error
  2. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning
  4. 2022Testing Stationarity and Change Point Detection in Reinforcement Learning
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning
  6. 2024Towards Adapting Reinforcement Learning Agents to New Tasks: Insights from Q-Values
  7. 2025FAST-Q: Fast-track Exploration with Adversarially Balanced State Representations for Counterfactual Action Estimation in Offline Reinforcement Learning
  8. 2024TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning