mlsanityData to 7 Oct 2026

Building a Subspace of Policies for Scalable Continual Learning

arXiv 2211.10445 ↗First posted 18 Nov 2022cs.LG cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

The ability to continuously acquire new knowledge and skills is crucial for autonomous agents. Existing methods are typically based on either fixed-size models that struggle to learn a large number of diverse behaviors, or growing-size models that scale poorly with the number of tasks. In this work, we aim to strike a better balance between an agent's size and performance by designing a method that grows adaptively depending on the task sequence. We introduce Continual Subspace of Policies (CSP), a new approach that incrementally builds a subspace of policies for training a reinforcement learning agent on a sequence of tasks. The subspace's high expressivity allows CSP to perform well for many different tasks while growing sublinearly with the number of tasks. Our method does not suffer from forgetting and displays positive transfer to new tasks. CSP outperforms a number of popular baselines on a wide range of scenarios from two challenging domains, Brax (locomotion) and Continual World (manipulation).

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 53 of its 66 references that are in the map. First posted on arXiv 18 Nov 2022.

Builds on

53 of 66 references.

References · 53

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016Layer Normalizationcited by 5,579
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2016Learning without Forgettingcited by 2,465
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2017Continual Learning with Deep Generative Replaycited by 1,152
  11. 2018Experience Replay for Continual Learningcited by 1,074
  12. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
38 more
  1. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  2. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  3. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  4. 2015Policy Distillationcited by 414
  5. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  6. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  7. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  8. 2018Selective Experience Replay for Lifelong Learningcited by 239
  9. 2020Supermasks in Superpositioncited by 223
  10. 2019Meta-Learning Representations for Continual Learningcited by 189
  11. 2018Reinforced Continual Learningcited by 186
  12. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  13. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 133
  14. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 112
  15. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  16. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 87
  17. 2020Learning to Continually Learncited by 82
  18. 2021Learning Neural Network Subspacescited by 82
  19. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 80
  20. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  21. 2019Superposition of many models into onecited by 64
  22. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  23. 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
  24. 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 42
  25. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 42
  26. 2020Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learningcited by 38
  27. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  28. 2019Policy Consolidation for Continual Reinforcement Learningcited by 37
  29. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 36
  30. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 35
  31. 2017Scalable Recollections for Continual Lifelong Learningcited by 33
  32. 2021Block Contextual MDPs for Continual Learningcited by 25
  33. 2020Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processescited by 23
  34. 2019Continual and Multi-Task Architecture Searchcited by 22
  35. 2020Online Meta-Critic Learning for Off-Policy Actor-Critic Methodscited by 17
  36. 2021CoMPS: Continual Meta Policy Searchcited by 15
  37. 2021Learning a subspace of policies for online adaptation in Reinforcement Learningcited by 14
  38. 2022Reinforcement Learning in Presence of Discrete Markovian Context Evolutioncited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 10; 2025: 8; 2026: 4.

28 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  3. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  4. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  5. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  6. 2025A Survey of Continual Reinforcement Learningcited by 16
  7. 2022The Effectiveness of World Models for Continual Reinforcement Learningcited by 15
  8. 2023Multi-agent Continual Coordination via Progressive Task Contextualizationcited by 10
  9. 2025Self-Composing Policies for Scalable Continual Reinforcement Learningcited by 10
  10. 2025Continual Reinforcement Learning by Planning with Online World Modelscited by 8
  11. 2024Multi-granularity Knowledge Transfer for Continual Reinforcement Learningcited by 6
  12. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
16 more
  1. 2024Identifying Policy Gradient Subspacescited by 5
  2. 2023Evaluating Continual Learning on a Home Robotcited by 4
  3. 2024Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environmentscited by 4
  4. 2025Continual Knowledge Adaptation for Reinforcement Learningcited by 3
  5. 2024I Know How: Combining Prior Policies to Solve New Taskscited by 2
  6. 2024Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representationscited by 2
  7. 2024Mitigating the Stability-Plasticity Dilemma in Adaptive Train Scheduling with Curriculum-Driven Continual DQN Expansioncited by 2
  8. 2024Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learningcited by 2
  9. 2025Mastering Continual Reinforcement Learning through Fine-Grained Sparse Network Allocation and Dormant Neuron Explorationcited by 2
  10. 2025Action-Adaptive Continual Learning: Enabling Policy Generalization under Dynamic Action Spacescited by 1
  11. 2026Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learningcited by 1
  12. 2024ResidualDroppath: Enhancing Feature Reuse over Residual Connectionscited by 0
  13. 2025FAST: Similarity-based Knowledge Transfer for Efficient Policy Learningcited by 0
  14. 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
  15. 2026From Pixels to Digital Agents: An Empirical Study on the Taxonomy and Technological Trends of Reinforcement Learning Environmentscited by 0
  16. 2026Efficient Exploration Is Enoughcited by 0