mlsanityData to 7 Oct 2026

Disentangling Transfer in Continual Reinforcement Learning

arXiv 2209.13900 ↗First posted 28 Sep 2022cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

The ability of continual learning systems to transfer knowledge from previously seen tasks in order to maximize performance on new tasks is a significant challenge for the field, limiting the applicability of continual learning solutions to realistic scenarios. Consequently, this study aims to broaden our understanding of transfer and its driving forces in the specific case of continual reinforcement learning. We adopt SAC as the underlying RL algorithm and Continual World as a suite of continuous control tasks. We systematically study how different components of SAC (the actor and the critic, exploration, and data) affect transfer efficacy, and we provide recommendations regarding various modeling options. The best set of choices, dubbed ClonEx-SAC, is evaluated on the recent Continual World benchmark. ClonEx-SAC achieves 87% final success rate compared to 80% of PackNet, the best method in the benchmark. Moreover, the transfer grows from 0.18 to 0.54 according to the metric provided by Continual World.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 28 Sep 2022.

Builds on

39 of 58 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
27 more
  1. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  2. 2018Experience Replay for Continual Learningcited by 1,074
  3. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  4. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  5. 2018A Survey on Deep Transfer Learningcited by 417
  6. 2020What is being transferred in transfer learning?cited by 365
  7. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  8. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  9. 2017Variational Continual Learningcited by 335
  10. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  11. 2019On Tiny Episodic Memories in Continual Learningcited by 299
  12. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  13. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  14. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
  15. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  16. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  17. 2018Towards Robust Evaluations of Continual Learningcited by 148
  18. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 124
  19. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  20. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  21. 2019A Survey of Deep Reinforcement Learning in Video Gamescited by 73
  22. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 42
  23. 2020Continual Model-Based Reinforcement Learning with Hypernetworkscited by 41
  24. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 35
  25. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
  26. 2018Continual Reinforcement Learning with Complex Synapsescited by 30
  27. 2020Hierarchical reinforcement learning for efficient exploration and transfercited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 6; 2025: 10; 2026: 7.

31 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2023Continual Learning: Applications and the Road Forwardcited by 59
  4. 2023Learning to Modulate pre-trained Models in RLcited by 28
  5. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  6. 2025A Survey of Continual Reinforcement Learningcited by 16
  7. 2026Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learningcited by 13
  8. 2023Continual Task Allocation in Meta-Policy Network via Sparse Promptingcited by 11
  9. 2025Self-Composing Policies for Scalable Continual Reinforcement Learningcited by 10
  10. 2023Replay-enhanced Continual Reinforcement Learningcited by 6
  11. 2024Multi-granularity Knowledge Transfer for Continual Reinforcement Learningcited by 6
  12. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
19 more
  1. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learningcited by 3
  2. 2024Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learningcited by 3
  3. 2025Efficient Multi-Task Reinforcement Learning with Cross-Task Policy Guidancecited by 3
  4. 2025Continual Knowledge Adaptation for Reinforcement Learningcited by 3
  5. 2025Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Explorationcited by 2
  6. 2023On the Value of Myopic Behavior in Policy Reusecited by 1
  7. 2024LoopSR: Looping Sim-and-Real for Lifelong Policy Adaptation of Legged Robotscited by 1
  8. 2024Continual Task Learning through Adaptive Policy Self-Compositioncited by 1
  9. 2026Continual Policy Consolidation for Lifelong Robot Learningcited by 1
  10. 2025Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learningcited by 0
  11. 2025Pay Attention to What and Where? Interpretable Feature Extractor in Vision-based Deep Reinforcement Learningcited by 0
  12. 2025Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systemscited by 0
  13. 2025Offline Behavioral Data Selectioncited by 0
  14. 2025Demonstration-Guided Continual Reinforcement Learning in Dynamic Environmentscited by 0
  15. 2026Learning Transferable Skills in Action RPGs via Directed Skill Graphs and Selective Adaptationcited by 0
  16. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  17. 2026Don't Forget the Critic: Value-Based Data Rehearsal for Multi-Cyclic Continual Reinforcement Learningcited by 0
  18. 2026Continual Quadruped Robots Coordination via Semantic Skill Discoverycited by 0
  19. 2026Offline Multi-agent Continual Cooperation via Skill Partition and Reusecited by 0