# Decision Boundary-aware Knowledge Consolidation Generates Better Instance-Incremental Learner

arXiv: https://arxiv.org/abs/2406.03065 · HTML: https://mlsanity.com/paper/2406.03065/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Placed in

- Knowledge distillation / [Distillation for continual learning](https://mlsanity.com/distill/distillation-for-continual-learning/)

## Builds on (most cited first, 23 of 23)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [CORe50: a New Dataset and Benchmark for Continuous Object Recognition](https://mlsanity.com/paper/1705.03550/) (2017)
- [Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines](https://mlsanity.com/paper/1810.12488/) (2018)
- [Class-incremental Learning via Deep Model Consolidation](https://mlsanity.com/paper/1903.07864/) (2019)
- [BREEDS: Benchmarks for Subpopulation Shift](https://mlsanity.com/paper/2008.04859/) (2020)
- [Adaptive Aggregation Networks for Class-Incremental Learning](https://mlsanity.com/paper/2010.05063/) (2020)
- [Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams](https://mlsanity.com/paper/2009.00919/) (2020)
- [Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation](https://mlsanity.com/paper/2204.00895/) (2022)
- [Lifelong GAN: Continual Learning for Conditional Image Generation](https://mlsanity.com/paper/1907.10107/) (2019)
- [Incremental Learning In Online Scenario](https://mlsanity.com/paper/2003.13191/) (2020)
- [Class-Incremental Learning with Strong Pre-trained Models](https://mlsanity.com/paper/2204.03634/) (2022)
- [Lifelong Person Re-Identification via Adaptive Knowledge Accumulation](https://mlsanity.com/paper/2103.12462/) (2021)
- [Online Prototype Learning for Online Continual Learning](https://mlsanity.com/paper/2308.00301/) (2023)
- [Exploring Example Influence in Continual Learning](https://mlsanity.com/paper/2209.12241/) (2022)
- [Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control](https://mlsanity.com/paper/1907.13322/) (2019)
- [A simple but strong baseline for online continual learning: Repeated Augmented Rehearsal](https://mlsanity.com/paper/2209.13917/) (2022)
- [Incremental Task Learning with Incremental Rank Updates](https://mlsanity.com/paper/2207.09074/) (2022)

## Built on by (most cited first, 1 of 1)

- [CLeAN: Continual Learning Adaptive Normalization in Dynamic Environments](https://mlsanity.com/paper/2603.17548/) (2026)

## Abstract (arXiv)

Instance-incremental learning (IIL) focuses on learning continually with data of the same classes. Compared to class-incremental learning (CIL), the IIL is seldom explored because IIL suffers less from catastrophic forgetting (CF). However, besides retaining knowledge, in real-world deployment scenarios where the class space is always predefined, continual and cost-effective model promotion with the potential unavailability of previous data is a more essential demand. Therefore, we first define a new and more practical IIL setting as promoting the model's performance besides resisting CF with only new observations. Two issues have to be tackled in the new IIL setting: 1) the notorious catastrophic forgetting because of no access to old data, and 2) broadening the existing decision boundary to new observations because of concept drift. To tackle these problems, our key insight is to moderately broaden the decision boundary to fail cases while retain old boundary. Hence, we propose a novel decision boundary-aware distillation method with consolidating knowledge to teacher to ease the student learning new knowledge. We also establish the benchmarks on existing datasets Cifar-100 and ImageNet. Notably, extensive experiments demonstrate that the teacher model can be a better incremental learner than the student model, which overturns previous knowledge distillation-based methods treating student as the main role.