# M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning

arXiv: https://arxiv.org/abs/1904.01769 · HTML: https://mlsanity.com/paper/1904.01769/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2019.

## Built on by (most cited first, 21 of 21)

- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge](https://mlsanity.com/paper/2103.01353/) (2021)
- [Memory Efficient Continual Learning with Transformers](https://mlsanity.com/paper/2203.04640/) (2022)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Contrastive Deep Supervision](https://mlsanity.com/paper/2207.05306/) (2022)
- [Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation](https://mlsanity.com/paper/2106.15287/) (2021)
- [Resolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learning](https://mlsanity.com/paper/2212.14284/) (2022)
- [Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants](https://mlsanity.com/paper/2308.01746/) (2023)
- [Generalized Knowledge Distillation via Relationship Matching](https://mlsanity.com/paper/2205.01915/) (2022)
- [Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory](https://mlsanity.com/paper/2309.15413/) (2023)
- [Dataset Knowledge Transfer for Class-Incremental Learning without Memory](https://mlsanity.com/paper/2110.08421/) (2021)
- [Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation](https://mlsanity.com/paper/2209.06107/) (2022)
- [DIODE: Dilatable Incremental Object Detection](https://mlsanity.com/paper/2108.05627/) (2021)
- [Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgery](https://mlsanity.com/paper/2402.05860/) (2024)
- [PlaStIL: Plastic and Stable Memory-Free Class-Incremental Learning](https://mlsanity.com/paper/2209.06606/) (2022)
- [Energy Aligning for Biased Models](https://mlsanity.com/paper/2106.03343/) (2021)
- [Online Continual Learning via the Knowledge Invariant and Spread-out Properties](https://mlsanity.com/paper/2302.00858/) (2023)
- [Improving Feature Generalizability with Multitask Learning in Class Incremental Learning](https://mlsanity.com/paper/2204.12915/) (2022)
- [EPIK: Eliminating multi-model Pipelines with Knowledge-distillation](https://mlsanity.com/paper/2211.14920/) (2022)
- [Realistic Continual Learning Approach using Pre-trained Models](https://mlsanity.com/paper/2404.07729/) (2024)

## Abstract (arXiv)

Incremental learning targets at achieving good performance on new categories without forgetting old ones. Knowledge distillation has been shown critical in preserving the performance on old classes. Conventional methods, however, sequentially distill knowledge only from the last model, leading to performance degradation on the old classes in later incremental learning steps. In this paper, we propose a multi-model and multi-level knowledge distillation strategy. Instead of sequentially distilling knowledge only from the last model, we directly leverage all previous model snapshots. In addition, we incorporate an auxiliary distillation to further preserve knowledge encoded at the intermediate feature levels. To make the model more memory efficient, we adapt mask based pruning to reconstruct all previous models with a small memory footprint. Experiments on standard incremental learning benchmarks show that our method preserves the knowledge on old classes better and improves the overall performance over standard distillation techniques.