mlsanityData to 7 Oct 2026

Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal

arXiv 2409.02512 ↗First posted 4 Sep 2024cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Artificial neural networks, especially recent diffusion-based models, have shown remarkable superiority in gaming, control, and QA systems, where the training tasks' datasets are usually static. However, in real-world applications, such as robotic control of reinforcement learning (RL), the tasks are changing, and new tasks arise in a sequential order. This situation poses the new challenge of plasticity-stability trade-off for training an agent who can adapt to task changes and retain acquired knowledge. In view of this, we propose a rehearsal-based continual diffusion model, called Continual Diffuser (CoD), to endow the diffuser with the capabilities of quick adaptation (plasticity) and lasting retention (stability). Specifically, we first construct an offline benchmark that contains 90 tasks from multiple domains. Then, we train the CoD on each task with sequential modeling and conditional generation for making decisions. Next, we preserve a small portion of previous datasets as the rehearsal buffer and replay it to retain the acquired knowledge. Extensive experiments on a series of tasks show CoD can achieve a promising plasticity-stability trade-off and outperform existing diffusion-based methods and other representative baselines on most tasks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 71 of its 83 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

71 of 83 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Solving Continual Offline Reinforcement Learning with Decision Transformer16 Jan
  2. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning21 Dec
  3. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 66

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
38 more
  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2017Continual Learning with Deep Generative Replaycited by 1,152
  6. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  7. 2018Experience Replay for Continual Learningcited by 1,074
  8. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  10. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  13. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  14. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  15. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  16. 2020Reinforcement Learning with Augmented Datacited by 452
  17. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  18. 2017Variational Continual Learningcited by 335
  19. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 211
  20. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  21. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
  22. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  23. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  24. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  25. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  26. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  27. 2023A Definition of Continual Reinforcement Learningcited by 98
  28. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  29. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  30. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  31. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RLcited by 65
  32. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  33. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  34. 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
  35. 2021A study on the plasticity of neural networkscited by 44
  36. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  37. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  38. 2019Policy Consolidation for Continual Reinforcement Learningcited by 37

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6
  2. 2024Continual Task Learning through Adaptive Policy Self-Compositioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay
  2. 2025Knowledge Retention for Continual Model-Based Reinforcement Learning
  3. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  4. 2023Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learning
  5. 2025Offline Reinforcement Learning with Discrete Diffusion Skills
  6. 2026TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning
  7. 2026State-Action Inpainting Diffuser for Continuous Control with Delay
  8. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learning