mlsanityData to 7 Oct 2026

Continual Deep Learning by Functional Regularisation of Memorable Past

arXiv 2004.14070 ↗First posted 29 Apr 2020stat.ML cs.LGcited by 104

Not on the map. Why?

Abstract (arXiv)

Continually learning new skills is important for intelligent systems, yet standard deep learning methods suffer from catastrophic forgetting of the past. Recent works address this with weight regularisation. Functional regularisation, although computationally expensive, is expected to perform better, but rarely does so in practice. In this paper, we fix this issue by using a new functional-regularisation approach that utilises a few memorable past examples crucial to avoid forgetting. By using a Gaussian Process formulation of deep networks, our approach enables training in weight-space while identifying both the memorable past and a functional prior. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation and memory-based methods are naturally combined.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 11; 2023: 24; 2024: 22; 2025: 16; 2026: 8.

104 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  3. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  4. 2024Continual Learning and Catastrophic Forgettingcited by 73
  5. 2022Representation Compensation Networks for Continual Semantic Segmentationcited by 72
  6. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  7. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  8. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 53
  9. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 47
  10. 2021Natural continual learning: success is a journey, not (just) a destinationcited by 46
  11. 2021Posterior Meta-Replay for Continual Learningcited by 45
  12. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
38 more
  1. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 42
  2. 2022Information-theoretic Online Memory Selection for Continual Learningcited by 42
  3. 2023A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Taskscited by 41
  4. 2023Tractable Function-Space Variational Inference in Bayesian Neural Networkscited by 41
  5. 2021Class-Incremental Learning with Generative Classifierscited by 39
  6. 2021Continually Learning Self-Supervised Representations with Projected Functional Regularizationcited by 38
  7. 2021Fast Adaptation with Linearized Neural Networkscited by 32
  8. 2023Continual Learning via Sequential Function-Space Variational Inferencecited by 28
  9. 2022Repairing Neural Networks by Leaving the Right Past Behindcited by 27
  10. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  11. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  12. 2021Formalizing the Generalization-Forgetting Trade-off in Continual Learningcited by 24
  13. 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
  14. 2023SIESTA: Efficient Online Continual Learning with Sleepcited by 24
  15. 2023How Efficient Are Today's Continual Learning Algorithms?cited by 24
  16. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  17. 2021Conditioning Sparse Variational Gaussian Processes for Online Decision-makingcited by 23
  18. 2023DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Cluescited by 23
  19. 2022Deep Learning and Symbolic Regression for Discovering Parametric Equationscited by 22
  20. 2023Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learningcited by 22
  21. 2021Knowledge-Adaptation Priorscited by 20
  22. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 20
  23. 2024A Unified and General Framework for Continual Learningcited by 19
  24. 2021Task-agnostic Continual Learning with Hybrid Probabilistic Modelscited by 16
  25. 2024Text-Enhanced Data-free Approach for Federated Class-Incremental Learningcited by 16
  26. 2021Lifelong Learning with Sketched Structural Regularizationcited by 15
  27. 2021Provable Continual Learning via Sketched Jacobian Approximationscited by 15
  28. 2024Investigating Critical Period Effects in Language Acquisition through Neural Language Modelscited by 15
  29. 2022Consistency is the key to further mitigating catastrophic forgetting in continual learningcited by 14
  30. 2022Lifelong Learning Metricscited by 13
  31. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  32. 2022Continual Variational Autoencoder Learning via Online Cooperative Memorizationcited by 12
  33. 2022Learn the Time to Learn: Replay Scheduling in Continual Learningcited by 11
  34. 2023Memory-Based Dual Gaussian Processes for Sequential Learningcited by 11
  35. 2024A Topology-aware Graph Coarsening Framework for Continual Graph Learningcited by 11
  36. 2024CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learningcited by 11
  37. 2025ProtoDepth: Unsupervised Continual Depth Completion with Prototypescited by 11
  38. 2024Don't Blame the Data, Blame the Model: Understanding Noise and Bias When Learning from Subjective Annotationscited by 10

50 most cited of 104 All 104 (JSON)