mlsanityData to 14 Oct 2026

Pausing Policy Learning in Non-stationary Reinforcement Learning

arXiv 2405.16053 ↗First posted 25 May 2024cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Real-time inference is a challenge of real-world reinforcement learning due to temporal differences in time-varying environments: the system collects data from the past, updates the decision model in the present, and deploys it in the future. We tackle a common belief that continually updating the decision is optimal to minimize the temporal gap. We propose forecasting an online reinforcement learning framework and show that strategically pausing decision updates yields better overall performance by effectively managing aleatoric uncertainty. Theoretically, we compute an optimal ratio between policy update and hold duration, and show that a non-zero policy hold duration provides a sharper upper bound on the dynamic regret. Our experimental evaluations on three different environments also reveal that a non-zero policy hold duration yields higher rewards compared to continuous decision updates.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 43 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

24 of 43 references.

References · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  3. 2023Mastering Diverse Domains through World Modelscited by 1,253
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2019Challenges of Real-World Reinforcement Learningcited by 291
  7. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  8. 2019Online Meta-Learningcited by 137
  9. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  10. 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 96
  11. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 91
  12. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 76
12 more
  1. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
  2. 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 48
  3. 2019Real-Time Reinforcement Learningcited by 40
  4. 2020Optimizing for the Future in Non-Stationary MDPscited by 37
  5. 2020Dynamic Regret of Policy Optimization in Non-stationary Environmentscited by 33
  6. 2022Factored Adaptation for Non-Stationary Reinforcement Learningcited by 30
  7. 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
  8. 2019Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Controlcited by 18
  9. 2020Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Controlcited by 17
  10. 2022Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraintscited by 10
  11. 2022Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, and Separation Designcited by 4
  12. 2023Tempo Adaptation in Non-stationary Reinforcement Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Predictive Control and Regret Analysis of Non-Stationary MDP with Look-ahead Informationcited by 2
  2. 2025Natural Policy Gradient for Average Reward Non-Stationary RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference
  2. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  3. 2025Forecasting in Offline Reinforcement Learning for Non-stationary Environments
  4. 2025Efficient Restarts in Non-Stationary Model-Free Reinforcement Learning
  5. 2026Model-Based Reinforcement Learning for Control under Time-Varying Dynamics
  6. 2026Dynamical Priors as a Training Objective in Reinforcement Learning
  7. 2025Rethinking the Foundations for Continual Reinforcement Learning
  8. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation