mlsanityData to 7 Oct 2026

Online Reinforcement Learning in Non-Stationary Context-Driven Environments

arXiv 2302.02182 ↗First posted 4 Feb 2023cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

We study online reinforcement learning (RL) in non-stationary environments, where a time-varying exogenous context process affects the environment dynamics. Online RL is challenging in such environments due to "catastrophic forgetting" (CF). The agent tends to forget prior knowledge as it trains on new experiences. Prior approaches to mitigate this issue assume task labels (which are often not available in practice), employ brittle regularization heuristics, or use off-policy methods that suffer from instability and poor performance. We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences while optimizing the return on current experiences. To perform this anchoring, LCPO locally constrains policy optimization using samples from experiences that lie outside of the current context distribution. We evaluate LCPO in Mujoco, classic control and computer systems environments with a variety of synthetic and real context traces, and find that it outperforms a variety of baselines in the non-stationary setting, while achieving results on-par with a "prescient" agent trained offline across all context traces. LCPO's source code is available at https://github.com/pouyahmdn/LCPO.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 72 of its 110 references that are in the map. First posted on arXiv 4 Feb 2023.

Builds on

72 of 110 references.

References · 72

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  11. 2016Progressive Neural Networkscited by 1,332
  12. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
38 more
  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2017Continual Learning with Deep Generative Replaycited by 1,152
  4. 2018Experience Replay for Continual Learningcited by 1,074
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  9. 2017Robust Adversarial Reinforcement Learningcited by 439
  10. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  12. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  13. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  14. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  15. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  16. 2018Selective Experience Replay for Lifelong Learningcited by 239
  17. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  18. 2018Task-Free Continual Learningcited by 219
  19. 2015Contextual Markov Decision Processescited by 213
  20. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  21. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
  22. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  23. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 159
  24. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  25. 2019Online Meta-Learningcited by 137
  26. 2018Learning Scheduling Algorithms for Data Processing Clusterscited by 133
  27. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  28. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
  29. 2021Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approachcited by 98
  30. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  31. 2020Deep Reinforcement Learning for Intelligent Transportation Systems: A Surveycited by 94
  32. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  33. 2016Learning values across many orders of magnitudecited by 80
  34. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
  35. 2019Neo: A Learned Query Optimizercited by 80
  36. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
  37. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 70
  38. 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 67

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  2. 2026Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learningcited by 4
  3. 2025Detecting Model Drifts in Non-Stationary Environment Using Edit Operation Measurescited by 0
  4. 2025Wavelet Predictive Representations for Non-Stationary Reinforcement Learningcited by 0
  5. 2026RASP-Tuner: Retrieval-Augmented Soft Prompts for Context-Aware Black-Box Optimization in Non-Stationary Environmentscited by 0
  6. 2026Online Change-point Detection for Cooperative Multi-Agent Reinforcement Learningcited by 0