mlsanityData to 14 Oct 2026

Using Curiosity for an Even Representation of Tasks in Continual Offline Reinforcement Learning

arXiv 2312.03177 ↗First posted 5 Dec 2023cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

In this work, we investigate the means of using curiosity on replay buffers to improve offline multi-task continual reinforcement learning when tasks, which are defined by the non-stationarity in the environment, are non labeled and not evenly exposed to the learner in time. In particular, we investigate the use of curiosity both as a tool for task boundary detection and as a priority metric when it comes to retaining old transition tuples, which we respectively use to propose two different buffers. Firstly, we propose a Hybrid Reservoir Buffer with Task Separation (HRBTS), where curiosity is used to detect task boundaries that are not known due to the task agnostic nature of the problem. Secondly, by using curiosity as a priority metric when it comes to retaining old transition tuples, a Hybrid Curious Buffer (HCB) is proposed. We ultimately show that these buffers, in conjunction with regular reinforcement learning algorithms, can be used to alleviate the catastrophic forgetting issue suffered by the state of the art on replay buffers when the agent's exposure to tasks is not equal along time. We evaluate catastrophic forgetting and the efficiency of our proposed buffers against the latest works such as the Hybrid Reservoir Buffer (HRB) and the Multi-Time Scale Replay Buffer (MTR) in three different continual reinforcement learning settings. Experiments were done on classical control tasks and Metaworld environment. Experiments show that our proposed replay buffers display better immunity to catastrophic forgetting compared to existing works in most of the settings.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 57 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

31 of 57 references.

References · 31

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  8. 2016OpenAI Gymcited by 1,736
  9. 2016Progressive Neural Networkscited by 1,361
  10. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
19 more
  1. 2018Experience Replay for Continual Learningcited by 1,095
  2. 2018DeepMind Control Suitecited by 956
  3. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  4. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  5. 2015Policy Distillationcited by 440
  6. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  7. 2018Selective Experience Replay for Lifelong Learningcited by 241
  8. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  9. 2018Don't forget, there is more than forgetting: new metrics for Continual Learningcited by 115
  10. 2013Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robotscited by 58
  11. 2019Policy Consolidation for Continual Reinforcement Learningcited by 39
  12. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 36
  13. 2019Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfercited by 26
  14. 2016VIME: Variational Information Maximizing Explorationcited by 16
  15. 2016Learning Shared Representations in Multi-task Reinforcement Learningcited by 15
  16. 2019Exploiting Hierarchy for Learning and Transfer in KL-regularized RLcited by 15
  17. 2018Computational Theories of Curiosity-Driven Learningcited by 11
  18. 2020Continual Reinforcement Learning with Multi-Timescale Replaycited by 10
  19. 2022Probing Transfer in Deep Reinforcement Learning without Task Engineeringcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Safe Continual Reinforcement Learning in Non-stationary Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learning
  2. 2026ARROW: Augmented Replay for RObust World models
  3. 2026TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning
  4. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay
  5. 2023CUDC: A Curiosity-Driven Unsupervised Data Collection Method with Adaptive Temporal Distances for Offline Reinforcement Learning
  6. 2024Augmenting Replay in World Models for Continual Reinforcement Learning
  7. 2022The Effectiveness of World Models for Continual Reinforcement Learning
  8. 2025An Empirical Study of Deep Reinforcement Learning in Continuing Tasks