mlsanityData to 14 Oct 2026

How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learning

arXiv 2610.03119 ↗First posted 2 Oct 2026cs.LG cs.AI cs.ROcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In lifelong reinforcement learning, retaining previously learned policies is not sufficient for effective transfer to a new task. Useful knowledge may be distributed across several prior policies, and its relevance may change as the learner acquires experience. One hypothesis is that task similarity can be effectively used in a continual learning setting to find and combine previously learned policies. To test it, Adaptive Mask Selection and Composition (AMSC) is designed to estimate similarity from online experience via non-parametric Wasserstein task embeddings from state-action-reward samples. The z-score-normalized sparsemax of the similarity scores are used to derive a variable-size support to periodically choose and weight policies to form a prior when learning a new task. On CT-graph and MiniGrid, AMSC achieves higher mean performance and forward transfer than the evaluated modular composition baselines while exhibiting no forgetting. Results on Continual World suggest that identifying relevant prior knowledge and determining its layer-specific composition may require additional layer-specific tuning. Ablations show that selecting relevant sources and determining how strongly to reuse them are central to these gains. Independently measured pairwise transfer is also positively associated with task-embedding similarity. These results indicate that task similarity can be an effective criterion to select and weight specific knowledge for reuse in lifelong reinforcement learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 49 of its 64 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

49 of 64 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Continual Knowledge Adaptation for Reinforcement Learning22 Oct
  2. 2025Self-Composing Policies for Scalable Continual Reinforcement Learning4 Jun
  3. 2025Lifelong Reinforcement Learning with Similarity-Driven Weighting by Large Models17 Mar
  4. 2024Statistical Context Detection for Deep Lifelong Reinforcement Learning29 May

Other references · 45

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2016Progressive Neural Networkscited by 1,361
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2018Experience Replay for Continual Learningcited by 1,095
  9. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  10. 2017Deep Reinforcement Learning that Matterscited by 754
  11. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  12. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
33 more
  1. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  2. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  3. 2019Continual learning with hypernetworkscited by 273
  4. 2018Selective Experience Replay for Lifelong Learningcited by 241
  5. 2020Supermasks in Superpositioncited by 227
  6. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  7. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 188
  8. 2020Geometric Dataset Distances via Optimal Transportcited by 158
  9. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
  10. 2022Jump-Start Reinforcement Learningcited by 112
  11. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 82
  12. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 81
  13. 2021Continual Learning via Local Module Compositioncited by 63
  14. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 60
  15. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 58
  16. 2022Modular Lifelong Reinforcement Learning via Neural Compositioncited by 40
  17. 2022Disentangling Transfer in Continual Reinforcement Learningcited by 33
  18. 2022Improving Task-free Continual Learning by Distributionally Robust Memory Evolutioncited by 31
  19. 2017Learning Wasserstein Embeddingscited by 26
  20. 2022Wasserstein Task Embedding for Measuring Task Similaritiescited by 25
  21. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 22
  22. 2022Continual Prune-and-Select: Class-incremental learning with specialized subnetworkscited by 17
  23. 2018Context-Aware Policy Reusecited by 16
  24. 2022Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualizationcited by 14
  25. 2021Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networkscited by 8
  26. 2021Mixture-of-Variational-Experts for Continual Learningcited by 8
  27. 2022Signing the Supermask: Keep, Hide, Invertcited by 7
  28. 2022Self-Activating Neural Ensembles for Continual Reinforcement Learningcited by 7
  29. 2023The configurable tree graph (CT-graph): measurable problems in partially observable and distal reward environments for lifelong reinforcement learningcited by 4
  30. 2023Continual Learning via Manifold Expansion Replaycited by 4
  31. 2023CHEEM: Continual Learning by Reuse, New, Adapt and Skip -- A Hierarchical Exploration-Exploitation Approachcited by 2
  32. 2025Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddingscited by 2
  33. 2025Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systemscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
  2. 2026To Retain or to Adapt? Generalizing Continual Learning
  3. 2026TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning
  4. 2024Continual Task Learning through Adaptive Policy Self-Composition
  5. 2025Continual Reinforcement Learning via Autoencoder-Driven Task and New Environment Recognition
  6. 2026Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment
  7. 2022The Effectiveness of World Models for Continual Reinforcement Learning
  8. 2026Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning