mlsanityData to 7 Oct 2026

Testing Stationarity and Change Point Detection in Reinforcement Learning

arXiv 2203.01707 ↗First posted 3 Mar 2022stat.ML cs.LGcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We consider offline reinforcement learning (RL) methods in possibly nonstationary environments. Many existing RL algorithms in the literature rely on the stationarity assumption that requires the system transition and the reward function to be constant over time. However, the stationarity assumption is restrictive in practice and is likely to be violated in a number of applications, including traffic signal control, robotics and mobile health. In this paper, we develop a consistent procedure to test the nonstationarity of the optimal Q-function based on pre-collected historical data, without additional online data collection. Based on the proposed test, we further develop a sequential change point detection method that can be naturally coupled with existing state-of-the-art RL methods for policy optimization in nonstationary environments. The usefulness of our method is illustrated by theoretical results, simulation studies, and a real data example from the 2018 Intern Health Study. A Python implementation of the proposed procedure is available at https://github.com/limengbinggz/CUSUM-RL.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 140 references that are in the map. First posted on arXiv 3 Mar 2022.

Builds on

42 of 140 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons26 Feb
  2. 2021Projected State-action Balancing Weights for Offline Reinforcement Learning10 Sep
  3. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inference6 Feb
  4. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  5. 2021Fast Rates for the Regret of Offline Reinforcement Learning31 Jan

Other references · 37

  1. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  2. 2019Batch Policy Learning under Constraintscited by 242
  3. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  4. 2021Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approachcited by 98
  5. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 75
  6. 2019Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activitycited by 70
  7. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  8. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  9. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  10. 2018Nearly Optimal Adaptive Procedure with Change Detection for Piecewise-Stationary Banditcited by 49
  11. 2019Learning When-to-Treat Policiescited by 49
  12. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
25 more
  1. 2019Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended versioncited by 36
  2. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  3. 2020Dynamic Regret of Policy Optimization in Non-stationary Environmentscited by 32
  4. 2020A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spacescited by 32
  5. 2015Optimal Sup-norm Rates and Uniform Inference on Nonlinear Functionals of Nonparametric IV Regressioncited by 30
  6. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  7. 2018Dynamic Assortment Optimization with Changing Contextual Informationcited by 28
  8. 2019Recovering Banditscited by 28
  9. 2016Assessing Time-Varying Causal Effect Moderation in Mobile Healthcited by 27
  10. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  11. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  12. 2020Reinforcement Learning for Optimization of COVID-19 Mitigation policiescited by 25
  13. 2016Multiple-change-point detection for high dimensional time series via sparsified binary segmentationcited by 24
  14. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  15. 2016High-dimensional changepoint estimation via sparse projectioncited by 20
  16. 2016Interpretable Dynamic Treatment Regimescited by 17
  17. 2021Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPscited by 16
  18. 2021The Micro-Randomized Trial for Developing Digital Interventions: Experimental Design and Data Analysis Considerationscited by 11
  19. 2020Personalized Policy Learning using Longitudinal Mobile Health Datacited by 10
  20. 2021Generalized non-stationary banditscited by 8
  21. 2021Bandit Quickest Changepoint Detectioncited by 7
  22. 2022Reinforcement Learning with Heterogeneous Data: Estimation and Inferencecited by 7
  23. 2016Multiscale and multilevel technique for consistent segmentation of nonstationary time seriescited by 5
  24. 2018Exponential inequalities for nonstationary Markov Chainscited by 3
  25. 2017Finite sample change point inference and identification for high-dimensional mean vectorscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 3; 2026: 2.

12 papers.

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2024A Review of Reinforcement Learning in Financial Applicationscited by 11
  3. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  4. 2026Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learningcited by 4
  5. 2022Doubly Inhomogeneous Reinforcement Learningcited by 3
  6. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  7. 2023Reinforcement Learning under External Influence: Guarantees, Algorithms, and Sample Complexitycited by 1
  8. 2023Life-inspired Interoceptive Artificial Intelligence for Autonomous and Adaptive Agentscited by 1
  9. 2024Introduction to Reinforcement Learningcited by 1
  10. 2025Detecting Model Drifts in Non-Stationary Environment Using Edit Operation Measurescited by 0
  11. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0
  12. 2026Online Change-point Detection for Cooperative Multi-Agent Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Forecasting in Offline Reinforcement Learning for Non-stationary Environments
  2. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity
  3. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning
  5. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints
  6. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios
  7. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  8. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning