mlsanityData to 7 Oct 2026

Data-Incremental Continual Offline Reinforcement Learning

arXiv 2404.12639 ↗First posted 19 Apr 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

In this work, we propose a new setting of continual learning: data-incremental continual offline reinforcement learning (DICORL), in which an agent is asked to learn a sequence of datasets of a single offline reinforcement learning (RL) task continually, instead of learning a sequence of offline RL tasks with respective datasets. Then, we propose that this new setting will introduce a unique challenge to continual learning: active forgetting, which means that the agent will forget the learnt skill actively. The main reason for active forgetting is conservative learning used by offline RL, which is used to solve the overestimation problem. With conservative learning, the offline RL method will suppress the value of all actions, learnt or not, without selection, unless it is in the just learning dataset. Therefore, inferior data may overlay premium data because of the learning sequence. To solve this problem, we propose a new algorithm, called experience-replay-based ensemble implicit Q-learning (EREIQL), which introduces multiple value networks to reduce the initial value and avoid using conservative learning, and the experience replay to relieve catastrophic forgetting. Our experiments show that EREIQL relieves active forgetting in DICORL and performs well.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 15 of its 36 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

15 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Solving Continual Offline Reinforcement Learning with Decision Transformer16 Jan

Other references · 14

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2018Experience Replay for Continual Learningcited by 1,074
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  8. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  9. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  10. 2017Continual Learning Through Synaptic Intelligencecited by 13
  11. 2021Selective Replay Enhances Learning in Online Continual Analogical Reasoningcited by 12
  12. 2020Continual Reinforcement Learning with Multi-Timescale Replaycited by 10
2 more
  1. 2022Learning to Predict Gradients for Semi-Supervised Continual Learningcited by 7
  2. 2021Brain-inspired feature exaggeration in generative replay for continual learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learning
  2. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay
  3. 2026TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning
  4. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
  5. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  6. 2021Offline Reinforcement Learning with Value-based Episodic Memory
  7. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  8. 2019An Optimistic Perspective on Offline Reinforcement Learning