# Amalgamating Knowledge towards Comprehensive Classification

arXiv: https://arxiv.org/abs/1811.02796 · HTML: https://mlsanity.com/paper/1811.02796/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2018.

## Built on by (most cited first, 30 of 38)

- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance](https://mlsanity.com/paper/2309.12530/) (2023)
- [Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data](https://mlsanity.com/paper/2110.15094/) (2021)
- [Training-Free Pretrained Model Merging](https://mlsanity.com/paper/2403.01753/) (2024)
- [Learning with Recoverable Forgetting](https://mlsanity.com/paper/2207.08224/) (2022)
- [Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks](https://mlsanity.com/paper/2109.12872/) (2021)
- [Distilling a Powerful Student Model via Online Knowledge Distillation](https://mlsanity.com/paper/2103.14473/) (2021)
- [Spot-adaptive Knowledge Distillation](https://mlsanity.com/paper/2205.02399/) (2022)
- [Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model](https://mlsanity.com/paper/2310.17653/) (2023)
- [KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation](https://mlsanity.com/paper/2105.04181/) (2021)
- [Safe Distillation Box](https://mlsanity.com/paper/2112.03695/) (2021)
- [Class-Incremental Learning via Knowledge Amalgamation](https://mlsanity.com/paper/2209.02112/) (2022)
- [Knowledge Distillation via Instance-level Sequence Learning](https://mlsanity.com/paper/2106.10885/) (2021)
- [VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition](https://mlsanity.com/paper/2408.16930/) (2024)
- [Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning](https://mlsanity.com/paper/2410.14633/) (2024)
- [Training Generative Adversarial Networks in One Stage](https://mlsanity.com/paper/2103.00430/) (2021)
- [Contrastive Knowledge Amalgamation for Unsupervised Image Classification](https://mlsanity.com/paper/2307.14781/) (2023)
- [Knowledge Amalgamation for Object Detection with Transformers](https://mlsanity.com/paper/2203.03187/) (2022)
- [HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models](https://mlsanity.com/paper/2506.19072/) (2025)
- [Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models](https://mlsanity.com/paper/2608.04349/) (2026)
- [Towards Communication-Efficient and Privacy-Preserving Federated Representation Learning](https://mlsanity.com/paper/2109.14611/) (2021)
- [Up to 100× Faster Data-free Knowledge Distillation](https://mlsanity.com/paper/2112.06253/) (2021)
- [Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Models](https://mlsanity.com/paper/2112.07327/) (2021)
- [Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal](https://mlsanity.com/paper/2305.00441/) (2023)
- [Multi-view Distillation based on Multi-modal Fusion for Few-shot Action Recognition(CLIP-M2DF)](https://mlsanity.com/paper/2401.08345/) (2024)
- [Optimizer Amalgamation](https://mlsanity.com/paper/2203.06474/) (2022)
- [Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks](https://mlsanity.com/paper/2203.13453/) (2022)
- [Federated Selective Aggregation for Knowledge Amalgamation](https://mlsanity.com/paper/2207.13309/) (2022)
- [From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models](https://mlsanity.com/paper/2210.05230/) (2022)
- [Self-Regulated Data-Free Knowledge Amalgamation for Text Classification](https://mlsanity.com/paper/2406.15476/) (2024)

## Abstract (arXiv)

With the rapid development of deep learning, there have been an unprecedentedly large number of trained deep network models available online. Reusing such trained models can significantly reduce the cost of training the new models from scratch, if not infeasible at all as the annotations used for the training original networks are often unavailable to public. We propose in this paper to study a new model-reusing task, which we term as knowledge amalgamation. Given multiple trained teacher networks, each of which specializes in a different classification problem, the goal of knowledge amalgamation is to learn a lightweight student model capable of handling the comprehensive classification. We assume no other annotations except the outputs from the teacher models are available, and thus focus on extracting and amalgamating knowledge from the multiple teachers. To this end, we propose a pilot two-step strategy to tackle the knowledge amalgamation task, by learning first the compact feature representations from teachers and then the network parameters in a layer-wise manner so as to build the student model. We apply this approach to four public datasets and obtain very encouraging results: even without any human annotation, the obtained student model is competent to handle the comprehensive classification task and in most cases outperforms the teachers in individual sub-tasks.