mlsanityData to 14 Oct 2026

Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning

arXiv 2207.05742 ↗First posted 12 Jul 2022cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

In lifelong learning, an agent learns throughout its entire life without resets, in a constantly changing environment, as we humans do. Consequently, lifelong learning comes with a plethora of research problems such as continual domain shifts, which result in non-stationary rewards and environment dynamics. These non-stationarities are difficult to detect and cope with due to their continuous nature. Therefore, exploration strategies and learning methods are required that are capable of tracking the steady domain shifts, and adapting to them. We propose Reactive Exploration to track and react to continual domain shifts in lifelong reinforcement learning, and to update the policy correspondingly. To this end, we conduct experiments in order to investigate different exploration strategies. We empirically show that representatives of the policy-gradient family are better suited for lifelong learning, as they adapt more quickly to distribution shifts than Q-learning. Thereby, policy-gradient methods profit the most from Reactive Exploration and show good results in lifelong learning with continual domain shifts. Our code is available at: https://github.com/ml-jku/reactive-exploration.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 58 of its 111 references that are in the map. First posted on arXiv 12 Jul 2022.

Builds on

58 of 111 references.

References · 58

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2016Learning without Forgettingcited by 2,496
  5. 2016OpenAI Gymcited by 1,736
  6. 2016Progressive Neural Networkscited by 1,361
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  8. 2015Prioritized Experience Replaycited by 1,099
  9. 2018Exploration by Random Network Distillationcited by 993
  10. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  11. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  12. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
38 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  2. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 364
  3. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  4. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  5. 2017Count-Based Exploration with Neural Density Modelscited by 257
  6. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  7. 2020Supermasks in Superpositioncited by 227
  8. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  9. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  10. 2019Meta-Learning Representations for Continual Learningcited by 189
  11. 2018Reinforced Continual Learningcited by 186
  12. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  13. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  14. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  15. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  16. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
  17. 2018Count-Based Exploration with the Successor Representationcited by 112
  18. 2018Continuous Learning in Single-Incremental-Task Scenarioscited by 109
  19. 2019Weighted Linear Bandits for Non-Stationary Environmentscited by 86
  20. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  21. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 76
  22. 2021A Simple Approach for Non-stationary Linear Banditscited by 54
  23. 2020La-MAML: Look-ahead Meta Learning for Continual Learningcited by 52
  24. 2018Nearly Optimal Adaptive Procedure with Change Detection for Piecewise-Stationary Banditcited by 50
  25. 2017Count-Based Exploration in Feature Space for Reinforcement Learningcited by 49
  26. 2021Bootstrapped Meta-Learningcited by 45
  27. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 43
  28. 2014Optimal Exploration-Exploitation in a Multi-Armed-Bandit Problem with Non-stationary Rewardscited by 42
  29. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  30. 2018Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organizationcited by 38
  31. 2020Optimizing for the Future in Non-Stationary MDPscited by 37
  32. 2019Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended versioncited by 36
  33. 2020Align-RUDDER: Learning From Few Demonstrations by Reward Redistributioncited by 36
  34. 2020Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updatescited by 34
  35. 2017Taming Non-stationary Bandits: A Bayesian Approachcited by 32
  36. 2020Cross-Domain Few-Shot Learning by Representation Fusioncited by 32
  37. 2018Learning Contextual Bandits in a Non-stationary Environmentcited by 26
  38. 2020Jelly Bean World: A Testbed for Never-Ending Learningcited by 19

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 3; 2025: 2; 2026: 2.

9 papers.

  1. 2023Learning to Modulate pre-trained Models in RLcited by 28
  2. 2025A Survey of Continual Reinforcement Learningcited by 18
  3. 2022The Effectiveness of World Models for Continual Reinforcement Learningcited by 17
  4. 2024Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayes Theorycited by 7
  5. 2026Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learningcited by 4
  6. 2024Contrastive Abstraction for Reinforcement Learningcited by 2
  7. 2025Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimizationcited by 2
  8. 2024Chaos-based reinforcement learning with TD3cited by 1
  9. 2026Online Change-point Detection for Cooperative Multi-Agent Reinforcement Learningcited by 0