# Class-Incremental Learning via Knowledge Amalgamation

arXiv: https://arxiv.org/abs/2209.02112 · HTML: https://mlsanity.com/paper/2209.02112/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 5 Sep 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Overcoming Catastrophic Forgetting by Incremental Moment Matching](https://mlsanity.com/paper/1703.08475/) (2017)
- [Using Hindsight to Anchor Past Knowledge in Continual Learning](https://mlsanity.com/paper/2002.08165/) (2020)
- [Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization](https://mlsanity.com/paper/1802.01569/) (2018)
- [Measuring and regularizing networks in function space](https://mlsanity.com/paper/1805.08289/) (2018)
- [Amalgamating Knowledge towards Comprehensive Classification](https://mlsanity.com/paper/1811.02796/) (2018)
- [Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning](https://mlsanity.com/paper/1906.10546/) (2019)
- [Embedded Encoder-Decoder in Convolutional Networks Towards Explainable AI](https://mlsanity.com/paper/2007.06712/) (2020)

## Built on by (most cited first, 9 of 9)

- [Training-Free Pretrained Model Merging](https://mlsanity.com/paper/2403.01753/) (2024)
- [Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model](https://mlsanity.com/paper/2310.17653/) (2023)
- [Few-Shot Continual Learning via Flat-to-Wide Approaches](https://mlsanity.com/paper/2306.14369/) (2023)
- [Few-Shot Class Incremental Learning via Robust Transformer Approach](https://mlsanity.com/paper/2405.05984/) (2024)
- [Continual Knowledge Consolidation LORA for Domain Incremental Learning](https://mlsanity.com/paper/2510.16077/) (2025)
- [Cross-Domain Continual Learning via CLAMP](https://mlsanity.com/paper/2405.07142/) (2024)
- [Unsupervised Few-Shot Continual Learning for Remote Sensing Image Scene Classification](https://mlsanity.com/paper/2406.18574/) (2024)
- [Source-Free Cross-Domain Continual Learning](https://mlsanity.com/paper/2510.01649/) (2025)
- [Few-Shot Domain Incremental Learning via Continual Vision-Language Consolidation](https://mlsanity.com/paper/2606.30190/) (2026)

## Abstract (arXiv)

Catastrophic forgetting has been a significant problem hindering the deployment of deep learning algorithms in the continual learning setting. Numerous methods have been proposed to address the catastrophic forgetting problem where an agent loses its generalization power of old tasks while learning new tasks. We put forward an alternative strategy to handle the catastrophic forgetting with knowledge amalgamation (CFA), which learns a student network from multiple heterogeneous teacher models specializing in previous tasks and can be applied to current offline methods. The knowledge amalgamation process is carried out in a single-head manner with only a selected number of memorized samples and no annotations. The teachers and students do not need to share the same network structure, allowing heterogeneous tasks to be adapted to a compact or sparse data representation. We compare our method with competitive baselines from different strategies, demonstrating our approach's advantages.