mlsanityData to 7 Oct 2026

Latent Spectral Regularization for Continual Learning

arXiv 2301.03345 ↗First posted 9 Jan 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

While biological intelligence grows organically as new knowledge is gathered throughout life, Artificial Neural Networks forget catastrophically whenever they face a changing training data distribution. Rehearsal-based Continual Learning (CL) approaches have been established as a versatile and reliable solution to overcome this limitation; however, sudden input disruptions and memory constraints are known to alter the consistency of their predictions. We study this phenomenon by investigating the geometric characteristics of the learner's latent space and find that replayed data points of different classes increasingly mix up, interfering with classification. Hence, we propose a geometric regularizer that enforces weak requirements on the Laplacian spectrum of the latent space, promoting a partitioning behavior. Our proposal, called Continual Spectral Regularizer for Incremental Learning (CaSpeR-IL), can be easily combined with any rehearsal-based CL approach and improves the performance of SOTA methods on standard benchmarks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 48 of its 66 references that are in the map. First posted on arXiv 9 Jan 2023.

Builds on

48 of 66 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2020Supervised Contrastive Learningcited by 2,803
  6. 2016Learning without Forgettingcited by 2,465
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  8. 2016Matching Networks for One Shot Learningcited by 2,246
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  11. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  12. 2021Learning to Prompt for Continual Learningcited by 938
36 more
  1. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  2. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  3. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  4. 2019Large Scale Incremental Learningcited by 727
  5. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  6. 2019Gradient based sample selection for online continual learningcited by 568
  7. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  8. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  9. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  10. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  11. 2021Co2L: Contrastive Continual Learningcited by 243
  12. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
  13. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  14. 2017Latent Space Oddity: on the Curvature of Deep Generative Modelscited by 169
  15. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 160
  16. 2018Towards Robust Evaluations of Continual Learningcited by 148
  17. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  18. 2020SS-IL: Separated Softmax for Incremental Learningcited by 126
  19. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
  20. 2020Rethinking Experience Replay: a Bag of Tricks for Continual Learningcited by 94
  21. 2017The Riemannian Geometry of Deep Generative Modelscited by 81
  22. 2015Partial Functional Correspondencecited by 61
  23. 2020LIMP: Learning Latent Shape Representations with Metric Preservation Priorscited by 54
  24. 2022Transfer without Forgettingcited by 44
  25. 2018Graph Coarsening with Preserved Spectral Propertiescited by 41
  26. 2021DualNet: Continual Learning, Fast and Slowcited by 39
  27. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  28. 2020Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgettingcited by 33
  29. 2020Class-incremental Learning with Pre-allocated Fixed Classifierscited by 33
  30. 2021Continual Semi-Supervised Learning through Contrastive Interpolation Consistencycited by 25
  31. 2018Isospectralization, or how to hear shape, style, and correspondencecited by 18
  32. 2022Class-Incremental Learning with Cross-Space Clustering and Controlled Transfercited by 15
  33. 2021Universal Spectral Adversarial Attacks for Deformable Shapescited by 14
  34. 2020Instant recovery of shape from spectrum via latent space connectionscited by 11
  35. 2020Representing Deep Neural Networks Latent Space Geometries with Graphscited by 7
  36. 2021Learning Spectral Unions of Partial Deformable 3D Shapescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 1; 2026: 2.

6 papers.

  1. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  2. 2023Wake-Sleep Consolidated Learningcited by 4
  3. 2024Class incremental learning with probability dampening and cascaded gated classifiercited by 1
  4. 2025Adaptive Model Ensemble for Continual Learningcited by 0
  5. 2026Dream2Learn: Structured Generative Dreaming for Continual Learningcited by 0
  6. 2026CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMscited by 0