mlsanityData to 14 Oct 2026

Continual learning with hypernetworks

arXiv 1906.00695 ↗First posted 3 Jun 2019cs.LG cs.AI stat.MLcited by 273

Not on the map. Why?

Abstract (arXiv)

Artificial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks. To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity. Continual learning (CL) is less difficult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-specific weight realizations, which can be maintained in memory using a simple regularizer. Besides achieving state-of-the-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories. Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable hypernetwork weights is comparable or smaller than target network size. We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning. Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.

Key facts. Cited by 273 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 45; 2022: 63; 2023: 63; 2024: 26; 2025: 42; 2026: 34.

273 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  3. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  4. 2021Personalized Federated Learning using Hypernetworkscited by 212
  5. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  6. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 200
  7. 2022Patching open-vocabulary models by interpolating weightscited by 190
  8. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 131
  9. 2021Repulsive Deep Ensembles are Bayesiancited by 104
  10. 2023A Brief Review of Hypernetworks in Deep Learningcited by 102
  11. 2021HyperInverter: Improving StyleGAN Inversion via Hypernetworkcited by 85
  12. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
38 more
  1. 2023Modular Deep Learningcited by 83
  2. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 81
  3. 2022A Theoretical Study on Solving Continual Learningcited by 80
  4. 2024Continual Learning and Catastrophic Forgettingcited by 77
  5. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 72
  6. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  7. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  8. 2022Re-basin via implicit Sinkhorn differentiationcited by 62
  9. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  10. 2025Text-to-LoRA: Instant Transformer Adaptioncited by 58
  11. 2022Pareto Set Learning for Neural Multi-objective Combinatorial Optimizationcited by 56
  12. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  13. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  14. 2021Efficient Feature Transformations for Discriminative and Generative Continual Learningcited by 50
  15. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 48
  16. 2021Natural continual learning: success is a journey, not (just) a destinationcited by 46
  17. 2021Posterior Meta-Replay for Continual Learningcited by 45
  18. 2022Learning Fast and Slow for Online Time Series Forecastingcited by 45
  19. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 43
  20. 2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
  21. 2021DualNet: Continual Learning, Fast and Slowcited by 39
  22. 2023MoStGAN-V: Video Generation with Temporal Motion Stylescited by 37
  23. 2021Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learningcited by 35
  24. 2021Recomposing the Reinforcement Learning Building Blocks with Hypernetworkscited by 35
  25. 2022DUET: A Tuning-Free Device-Cloud Collaborative Parameters Generation Framework for Efficient Device Model Generalizationcited by 35
  26. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  27. 2021Continual Learning for Recurrent Neural Networks: an Empirical Evaluationcited by 34
  28. 2023Hyper-Decision Transformer for Efficient Online Policy Adaptationcited by 33
  29. 2022Continual Learning with Recursive Gradient Optimizationcited by 32
  30. 2022Improving Task-free Continual Learning by Distributionally Robust Memory Evolutioncited by 31
  31. 2021Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffercited by 30
  32. 2021CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Taskscited by 29
  33. 2023Learnability and Algorithm for Continual Learningcited by 28
  34. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  35. 2022A Multi-Head Model for Continual Learning via Out-of-Distribution Replaycited by 27
  36. 2023HyperDeepONet: learning operator with complex target function space using the limited resources via hypernetworkcited by 26
  37. 2021Optimizing Reusable Knowledge for Continual Learning via Metalearningcited by 25
  38. 2022HyperTime: Implicit Neural Representation for Time Seriescited by 24

50 most cited of 273 All 273 (JSON)