mlsanityData to 7 Oct 2026

Continual Learning in Deep Networks: an Analysis of the Last Layer

arXiv 2106.01834 ↗First posted 3 Jun 2021cs.LG cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

We study how different output layer parameterizations of a deep neural network affects learning and forgetting in continual learning settings. The following three effects can cause catastrophic forgetting in the output layer: (1) weights modifications, (2) interference, and (3) projection drift. In this paper, our goal is to provide more insights into how changing the output layer parameterization may address (1) and (2). Some potential solutions to those issues are proposed and evaluated here in several continual learning scenarios. We show that the best-performing type of output layer depends on the data distribution drifts and/or the amount of data available. In particular, in some cases where a standard linear layer would fail, changing parameterization is sufficient to achieve a significantly better performance, without introducing any continual-learning algorithm but instead by using standard SGD to train a model. Our analysis and results shed light on the dynamics of the output layer in continual learning scenarios and suggest a way of selecting the best type of output layer for a given scenario.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

40 of 64 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2016Learning without Forgettingcited by 2,465
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  9. 2017Continual Learning with Deep Generative Replaycited by 1,152
  10. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  11. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  12. 2019Large Scale Incremental Learningcited by 727
28 more
  1. 2019Three scenarios for continual learningcited by 610
  2. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  3. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  4. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  5. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  6. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  7. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 281
  8. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  9. 2020Semantic Drift Compensation for Class-Incremental Learningcited by 241
  10. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  11. 2019Meta-Learning Representations for Continual Learningcited by 189
  12. 2018Towards Robust Evaluations of Continual Learningcited by 148
  13. 2019Continual Unsupervised Representation Learningcited by 147
  14. 2018State Representation Learning for Control: An Overviewcited by 125
  15. 2019Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysiscited by 91
  16. 2020Optimal Continual Learning has Perfect Memory and is NP-hardcited by 86
  17. 2018Generative Models from the perspective of Continual Learningcited by 69
  18. 2019Rehearsal-Free Continual Learning over Small Non-I.I.D. Batchescited by 46
  19. 2020Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learningcited by 38
  20. 2021Understanding Continual Learning Settings with Data Distribution Drift Analysiscited by 38
  21. 2021Continuum: Simple Management of Complex Continual Learning Scenarioscited by 33
  22. 2018Incremental Classifier Learning with Generative Adversarial Networkscited by 32
  23. 2019Regularization Shortcomings for Continual Learningcited by 32
  24. 2018DeeSIL: Deep-Shallow Incremental Learningcited by 30
  25. 2019Continual Learning in Practicecited by 28
  26. 2019Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based roboticscited by 21
  27. 2020Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processescited by 17
  28. 2018Marginal Replay vs Conditional Replay for Continual Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 5; 2024: 2; 2025: 1; 2026: 1.

17 papers.

Sort:

  1. 2022Constrained Few-shot Class-incremental Learningcited by 93
  2. 2021Continual Learning via Local Module Compositioncited by 61
  3. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  4. 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
  5. 2022Continual Learning Beyond a Single Modelcited by 18
  6. 2023Does Continual Learning Equally Forget All Parameters?cited by 18
  7. 2022Beyond Supervised Continual Learning: a Reviewcited by 11
  8. 2022Continual Feature Selection: Spurious Features in Continual Learningcited by 7
  9. 2022Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentationcited by 4
  10. 2023SHARP: Sparsity and Hidden Activation RePlay for Neuro-Inspired Continual Learningcited by 1
  11. 2023Continual Learning with Pretrained Backbones by Tuning in the Input Spacecited by 1
  12. 2023Diagnosing Catastrophe: Large parts of accuracy loss in continual learning can be accounted for by readout misalignmentcited by 1
5 more
  1. 2023TLCE: Transfer-Learning Based Classifier Ensembles for Few-Shot Class-Incremental Learningcited by 0
  2. 2024Continual Learning by Three-Phase Consolidationcited by 0
  3. 2024Task agnostic continual learning with Pairwise layer architecturecited by 0
  4. 2025Hybrid Learners Do Not Forget: A Brain-Inspired Neuro-Symbolic Approach to Continual Learningcited by 0
  5. 2026FreeMOCA: Memory-Free Continual Learning for Malicious Code Analysiscited by 0