mlsanityData to 14 Oct 2026

S2IL: Structurally Stable Incremental Learning

arXiv 2503.12193 ↗First posted 15 Mar 2025cs.CV cs.LGcited by 0

On the map: Knowledge distillation → Distillation for continual learning

Abstract (arXiv)

Feature Distillation (FD) strategies are proven to be effective in mitigating Catastrophic Forgetting (CF) seen in Class Incremental Learning (CIL). However, current FD approaches enforce strict alignment of feature magnitudes and directions across incremental steps, limiting the model's ability to adapt to new knowledge. In this paper we propose Structurally Stable Incremental Learning(S22IL), a FD method for CIL that mitigates CF by focusing on preserving the overall spatial patterns of features which promote flexible (plasticity) yet stable representations that preserve old knowledge (stability). We also demonstrate that our proposed method S2IL achieves strong incremental accuracy and outperforms other FD methods on SOTA benchmark datasets CIFAR-100, ImageNet-100 and ImageNet-1K. Notably, S2IL outperforms other methods by a significant margin in scenarios that have a large number of incremental tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation. Builds on 24 of its 38 references that are in the map. First posted on arXiv 15 Mar 2025.

Builds on

24 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation2 Apr

Other references · 23

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2016Learning without Forgettingcited by 2,496
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  5. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  6. 2016Progressive Neural Networkscited by 1,361
  7. 2017Continual Learning with Deep Generative Replaycited by 1,158
  8. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  10. 2019Large Scale Incremental Learningcited by 732
  11. 2019Three scenarios for continual learningcited by 614
  12. 2019Gradient based sample selection for online continual learningcited by 571
11 more
  1. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  2. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  3. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  4. 2020Adaptive Aggregation Networks for Class-Incremental Learningcited by 149
  5. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  6. 2023On the Stability-Plasticity Dilemma of Class-Incremental Learningcited by 56
  7. 2022Class-Incremental Learning for Action Recognition in Videoscited by 46
  8. 2024Gradient Reweighting: Towards Imbalanced Class-Incremental Learningcited by 37
  9. 2023Exploring Semantic Consistency in Unpaired Image Translation to Generate Data for Surgical Applicationscited by 9
  10. 2024EXACFS -- A CIL Method to mitigate Catastrophic Forgettingcited by 2
  11. 2024CLFace: A Scalable and Resource-Efficient Continual Learning Framework for Lifelong Face Recognitioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning
  2. 2022PlaStIL: Plastic and Stable Memory-Free Class-Incremental Learning
  3. 2024On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning
  4. 2023Fine-Grained Knowledge Selection and Restoration for Non-Exemplar Class Incremental Learning
  5. 2021On Learning the Geodesic Path for Incremental Learning
  6. 2023SRIL: Selective Regularization for Class-Incremental Learning
  7. 2021In Defense of the Learning Without Forgetting for Task Incremental Learning
  8. 2023Class-Incremental Learning using Diffusion Model for Distillation and Replay