mlsanityData to 7 Oct 2026

Continual Task Learning through Adaptive Policy Self-Composition

arXiv 2411.11364 ↗First posted 18 Nov 2024cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Training a generalizable agent to continually learn a sequence of tasks from offline trajectories is a natural requirement for long-lived agents, yet remains a significant challenge for current offline reinforcement learning (RL) algorithms. Specifically, an agent must be able to rapidly adapt to new tasks using newly collected trajectories (plasticity), while retaining knowledge from previously learned tasks (stability). However, systematic analyses of this setting are scarce, and it remains unclear whether conventional continual learning (CL) methods are effective in continual offline RL (CORL) scenarios. In this study, we develop the Offline Continual World benchmark and demonstrate that traditional CL methods struggle with catastrophic forgetting, primarily due to the unique distribution shifts inherent to CORL scenarios. To address this challenge, we introduce CompoFormer, a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Upon encountering a new task, CompoFormer leverages semantic correlations to selectively integrate relevant prior policies alongside newly trained parameters, thereby enhancing knowledge sharing and accelerating the learning process. Our experiments reveal that CompoFormer outperforms conventional CL methods, particularly in longer task sequences, showcasing a promising balance between plasticity and stability.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 49 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

45 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal4 Sep
  2. 2024HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning28 May
  3. 2024Q-value Regularized Transformer for Offline Reinforcement Learning27 May
  4. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?20 May
  5. 2024Solving Continual Offline Reinforcement Learning with Decision Transformer16 Jan
  6. 2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policies14 Mar
  7. 2023Graph Decision Transformer7 Mar
  8. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 36

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2016Learning without Forgettingcited by 2,465
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2016Progressive Neural Networkscited by 1,332
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
24 more
  1. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  2. 2018Experience Replay for Continual Learningcited by 1,074
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  6. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  7. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  8. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  9. 2017Variational Continual Learningcited by 335
  10. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  11. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  12. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  13. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  14. 2024Learning Multi-Agent Communication from Graph Modeling Perspectivecited by 47
  15. 2020Interference and Generalization in Temporal Difference Learningcited by 41
  16. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  17. 2022Disentangling Transfer in Continual Reinforcement Learningcited by 31
  18. 2018Mix&Match - Agent Curricula for Reinforcement Learningcited by 26
  19. 2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
  20. 2023Continual Task Allocation in Meta-Policy Network via Sparse Promptingcited by 11
  21. 2023Balanced Destruction-Reconstruction Dynamics for Memory-replay Class Incremental Learningcited by 10
  22. 2022Learning to Learn: How to Continuously Teach Humans and Machinescited by 9
  23. 2024t-DGR: A Trajectory-Based Deep Generative Replay Method for Continual Learning in Decision Makingcited by 9
  24. 2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalizationcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Continual Knowledge Adaptation for Reinforcement Learningcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning
  2. 2026Benchmarking World Models for Continual Learning on Compositional Tasks
  3. 2025Continual Reinforcement Learning by Planning with Online World Models
  4. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learning
  5. 2026RL Forgets! Towards Continual Policy Optimization
  6. 2024Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learning
  7. 2026When Does Continual Learning Require Learning
  8. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation