# Class-incremental Learning via Deep Model Consolidation

arXiv: https://arxiv.org/abs/1903.07864 · HTML: https://mlsanity.com/paper/1903.07864/

## Key facts

- Cited by 173 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2019.

## Built on by (most cited first, 30 of 173)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need](https://mlsanity.com/paper/2303.07338/) (2023)
- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2103.04059/) (2021)
- [TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation](https://mlsanity.com/paper/2303.06937/) (2023)
- [InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions](https://mlsanity.com/paper/2403.11435/) (2024)
- [ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection](https://mlsanity.com/paper/2205.14922/) (2022)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Self-Supervised Training Enhances Online Continual Learning](https://mlsanity.com/paper/2103.14010/) (2021)
- [An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions](https://mlsanity.com/paper/2204.08817/) (2022)
- [Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning](https://mlsanity.com/paper/2403.18886/) (2024)
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](https://mlsanity.com/paper/2208.11296/) (2022)
- [Efficient Feature Transformations for Discriminative and Generative Continual Learning](https://mlsanity.com/paper/2103.13558/) (2021)
- [Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusion](https://mlsanity.com/paper/2407.14143/) (2024)
- [Memory Efficient Continual Learning with Transformers](https://mlsanity.com/paper/2203.04640/) (2022)
- [Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning](https://mlsanity.com/paper/2303.09483/) (2023)
- [When Prompt-based Incremental Learning Does Not Meet Strong Pretraining](https://mlsanity.com/paper/2308.10445/) (2023)
- [Parameter-Level Soft-Masking for Continual Learning](https://mlsanity.com/paper/2306.14775/) (2023)
- [Multi-domain Learning for Updating Face Anti-spoofing Models](https://mlsanity.com/paper/2208.11148/) (2022)
- [Multi-View Correlation Distillation for Incremental Object Detection](https://mlsanity.com/paper/2107.01787/) (2021)
- [Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2407.05342/) (2024)
- [Sample Condensation in Online Continual Learning](https://mlsanity.com/paper/2206.11849/) (2022)
- [Class-Incremental Learning using Diffusion Model for Distillation and Replay](https://mlsanity.com/paper/2306.17560/) (2023)
- [Balancing Stability and Plasticity through Advanced Null Space in Continual Learning](https://mlsanity.com/paper/2207.12061/) (2022)
- [Meta-attention for ViT-backed Continual Learning](https://mlsanity.com/paper/2203.11684/) (2022)
- [Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation](https://mlsanity.com/paper/2211.06866/) (2022)
- [Continual Semi-Supervised Learning through Contrastive Interpolation Consistency](https://mlsanity.com/paper/2108.06552/) (2021)
- [SATS: Self-Attention Transfer for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.07667/) (2022)

## Abstract (arXiv)

Deep neural networks (DNNs) often suffer from "catastrophic forgetting" during incremental learning (IL) --- an abrupt degradation of performance on the original set of classes when the training objective is adapted to a newly added set of classes. Existing IL approaches tend to produce a model that is biased towards either the old classes or new classes, unless with the help of exemplars of the old data. To address this issue, we propose a class-incremental learning paradigm called Deep Model Consolidation (DMC), which works well even when the original training data is not available. The idea is to first train a separate model only for the new classes, and then combine the two individual models trained on data of two distinct set of classes (old classes and new classes) via a novel double distillation training objective. The two existing models are consolidated by exploiting publicly available unlabeled auxiliary data. This overcomes the potential difficulties due to the unavailability of original training data. Compared to the state-of-the-art techniques, DMC demonstrates significantly better performance in image classification (CIFAR-100 and CUB-200) and object detection (PASCAL VOC 2007) in the single-headed IL setting.