mlsanityData to 14 Oct 2026

An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Context

arXiv 2212.12735 ↗First posted 24 Dec 2022cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

One of the key challenges in deploying RL to real-world applications is to adapt to variations of unknown environment contexts, such as changing terrains in robotic tasks and fluctuated bandwidth in congestion control. Existing works on adaptation to unknown environment contexts either assume the contexts are the same for the whole episode or assume the context variables are Markovian. However, in many real-world applications, the environment context usually stays stable for a stochastic period and then changes in an abrupt and unpredictable manner within an episode, resulting in a segment structure, which existing works fail to address. To leverage the segment structure of piecewise stable context in real-world applications, in this paper, we propose a textitSegmented Context Belief Augmented Deep~(SeCBAD) RL method. Our method can jointly infer the belief distribution over latent context with the posterior over segment length and perform more accurate belief context inference with observed data within the current context segment. The inferred belief context can be leveraged to augment the state, leading to a policy that can adapt to abrupt variations in context. We demonstrate empirically that SeCBAD can infer context segment length accurately and outperform existing methods on a toy grid world environment and Mujuco tasks with piecewise-stable context.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 24 Dec 2022.

Builds on

32 of 43 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2018World Modelscited by 1,513
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  10. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  11. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  12. 2015Contextual Markov Decision Processescited by 220
20 more
  1. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  2. 2015Deep Kalman Filterscited by 166
  3. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 156
  4. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  5. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 81
  6. 2019Meta reinforcement learning as task inferencecited by 77
  7. 2018Probabilistic Recurrent State-Space Modelscited by 64
  8. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  9. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 47
  10. 2020Optimizing for the Future in Non-Stationary MDPscited by 37
  11. 2022Factored Adaptation for Non-Stationary Reinforcement Learningcited by 30
  12. 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
  13. 2021Block Contextual MDPs for Continual Learningcited by 25
  14. 2020MELD: Meta-Reinforcement Learning from Images via Latent State Modelscited by 22
  15. 2021Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detectioncited by 18
  16. 2022Robust Policy Learning over Multiple Uncertainty Setscited by 18
  17. 2020Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularizationcited by 16
  18. 2021Bayesian Bellman Operatorscited by 16
  19. 2022Reinforcement Learning in Presence of Discrete Markovian Context Evolutioncited by 9
  20. 2021Meta-Reinforcement Learning by Tracking Task Non-stationaritycited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 4; 2026: 4.

15 papers.

Sort:

By area:

  1. 2023Reinforcement Learning with History-Dependent Dynamic Contextscited by 11
  2. 2023Online Reinforcement Learning in Non-Stationary Context-Driven Environmentscited by 6
  3. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationaritycited by 5
  4. 2024DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learningcited by 4
  5. 2023Tempo Adaptation in Non-stationary Reinforcement Learningcited by 3
  6. 2024GRAM: Generalization in Deep RL with a Robust Adaptation Modulecited by 3
  7. 2025Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysiscited by 3
  8. 2025Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shiftscited by 2
  9. 2023Anytime-Competitive Reinforcement Learning with Policy Priorcited by 1
  10. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 1
  11. 2025Wavelet Predictive Representations for Non-Stationary Reinforcement Learningcited by 0
  12. 2025Interpretable Hybrid Deep Q-Learning Framework for IoT-Based Food Spoilage Prediction with Synthetic Data Generation and Hardware Validationcited by 0
3 more
  1. 2026The Cost of Learning Under Multiple Change Pointscited by 0
  2. 2026Post-Hoc Robustness for Model-Based Reinforcement Learningcited by 0
  3. 2026Changepoint-Aware World Models: Detecting Dynamics Shifts and Recovering by Forgetting Stale Replay in Model-Based RLcited by 0