mlsanityData to 14 Oct 2026

Self-Composing Policies for Scalable Continual Reinforcement Learning

arXiv 2506.14811 ↗First posted 4 Jun 2025cs.LGcited by 11

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

This work introduces a growable and modular neural network architecture that naturally avoids catastrophic forgetting and interference in continual reinforcement learning. The structure of each module allows the selective combination of previous policies along with its internal policy, accelerating the learning process on the current task. Unlike previous growing neural network approaches, we show that the number of parameters of the proposed approach grows linearly with respect to the number of tasks, and does not sacrifice plasticity to scale. Experiments conducted in benchmark continuous control and visual problems reveal that the proposed approach achieves greater knowledge transfer and performance than alternative methods.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 38 of its 55 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

38 of 55 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2016Progressive Neural Networkscited by 1,361
  10. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  11. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
26 more
  1. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  2. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  3. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  4. 2015Policy Distillationcited by 440
  5. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  6. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  7. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  8. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  9. 2017Automated Curriculum Learning for Neural Networkscited by 257
  10. 2020Supermasks in Superpositioncited by 227
  11. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  12. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  13. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  14. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  15. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
  16. 2018Don't forget, there is more than forgetting: new metrics for Continual Learningcited by 115
  17. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 94
  18. 2019Routing Networks and the Challenges of Modular and Compositional Computationcited by 50
  19. 2022Modular Lifelong Reinforcement Learning via Neural Compositioncited by 40
  20. 2022Disentangling Transfer in Continual Reinforcement Learningcited by 33
  21. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  22. 2018Mix&Match - Agent Curricula for Reinforcement Learningcited by 27
  23. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  24. 2019Knowledge transfer in deep block-modular neural networkscited by 14
  25. 2021Toward Robust Long Range Policy Transfercited by 6
  26. 2018On the role of neurogenesis in overcoming catastrophic forgettingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025A Survey of Continual Reinforcement Learningcited by 18
  2. 2025Continual Knowledge Adaptation for Reinforcement Learningcited by 5
  3. 2025Policy Compatible Skill Incremental Learning via Lazy Learning Interfacecited by 3
  4. 2026Information-Theoretic Constraints for Continual Vision-Language-Action Alignmentcited by 3
  5. 2025AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Researchcited by 2
  6. 2025Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environmentscited by 1
  7. 2025Dynamic Mixture of Experts Against Severe Distribution Shiftscited by 0
  8. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  9. 2026Efficient Exploration Is Enoughcited by 0
  10. 2026CARVE: Breaking Data Barriers in Chip Placement by Harnessing Reusable Expertisecited by 0
  11. 2026How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Hierarchical Orchestra of Policies
  2. 2025Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Exploration
  3. 2024Continual Task Learning through Adaptive Policy Self-Composition
  4. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
  5. 2024Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learning
  6. 2025Advancements and Challenges in Continual Reinforcement Learning: A Comprehensive Review
  7. 2026TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning
  8. 2024Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning