mlsanityData to 14 Oct 2026

Supermasks in Superposition

arXiv 2006.14769 ↗First posted 26 Jun 2020cs.LG cs.AI stat.MLcited by 227

Not on the map. Why?

Abstract (arXiv)

We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting. Our approach uses a randomly initialized, fixed base network and for each task finds a subnetwork (supermask) that achieves good performance. If task identity is given at test time, the correct subnetwork can be retrieved with minimal memory usage. If not provided, SupSup can infer the task using gradient-based optimization to find a linear superposition of learned supermasks which minimizes the output entropy. In practice we find that a single gradient step is often sufficient to identify the correct mask, even among 2500 tasks. We also showcase two promising extensions. First, SupSup models can be trained entirely without task identity information, as they may detect when they are uncertain about new data and allocate an additional supermask for the new training distribution. Finally the entire, growing set of supermasks can be stored in a constant-sized reservoir by implicitly storing them as attractors in a fixed-sized Hopfield network.

Key facts. Cited by 227 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 28; 2022: 46; 2023: 57; 2024: 46; 2025: 34; 2026: 16.

227 papers.

Sort:

By area:

  1. 2021Learning to Prompt for Continual Learningcited by 944
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  3. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  4. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  5. 2023Continual Pre-training of Language Modelscited by 198
  6. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 197
  7. 2022Patching open-vocabulary models by interpolating weightscited by 190
  8. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  9. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  10. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  11. 2023Modular Deep Learningcited by 83
  12. 2022A Theoretical Study on Solving Continual Learningcited by 80
38 more
  1. 2024Continual Learning and Catastrophic Forgettingcited by 77
  2. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  3. 2021Wide Neural Networks Forget Less Catastrophicallycited by 64
  4. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  5. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 61
  6. 2022Class-Incremental Learning with Strong Pre-trained Modelscited by 60
  7. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  8. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  9. 2021Efficient Feature Transformations for Discriminative and Generative Continual Learningcited by 50
  10. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 50
  11. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  12. 2023Introducing Language Guidance in Prompt-based Continual Learningcited by 49
  13. 2022Architecture Matters in Continual Learningcited by 46
  14. 2023AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learningcited by 46
  15. 2021Posterior Meta-Replay for Continual Learningcited by 45
  16. 2022PIVOT: Prompting for Video Continual Learningcited by 45
  17. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  18. 2022Memory Efficient Continual Learning with Transformerscited by 43
  19. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
  20. 2024Incremental Residual Concept Bottleneck Modelscited by 43
  21. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  22. 2023Parameter-Level Soft-Masking for Continual Learningcited by 40
  23. 2022On the Soft-Subnetwork for Few-shot Class Incremental Learningcited by 37
  24. 2021Hierarchical Prototype Networks for Continual Graph Representation Learningcited by 28
  25. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  26. 2023Learnability and Algorithm for Continual Learningcited by 28
  27. 2021Kernel Continual Learningcited by 27
  28. 2022A Multi-Head Model for Continual Learning via Out-of-Distribution Replaycited by 27
  29. 2021Optimizing Reusable Knowledge for Continual Learning via Metalearningcited by 25
  30. 2021Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environmentscited by 25
  31. 2023ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysiscited by 25
  32. 2023Overcoming Generic Knowledge Loss with Selective Parameter Updatecited by 23
  33. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 22
  34. 2023Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learningcited by 22
  35. 2022Continual Learning Based on OOD Detection and Task Maskingcited by 21
  36. 2024Continual Learning on Graphs: Challenges, Solutions, and Opportunitiescited by 21
  37. 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
  38. 2023Continual Learning of Diffusion Models with Generative Distillationcited by 20

50 most cited of 227 All 227 (JSON)