# Contrastive Knowledge Amalgamation for Unsupervised Image Classification

arXiv: https://arxiv.org/abs/2307.14781 · HTML: https://mlsanity.com/paper/2307.14781/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 27 Jul 2023.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Amalgamating Knowledge towards Comprehensive Classification](https://mlsanity.com/paper/1811.02796/) (2018)
- [Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning](https://mlsanity.com/paper/1906.10546/) (2019)
- [Unifying Heterogeneous Classifiers with Distillation](https://mlsanity.com/paper/1904.06062/) (2019)
- [PTMTorrent: A Dataset for Mining Open-source Pre-trained Model Packages](https://mlsanity.com/paper/2303.08934/) (2023)

## Built on by (most cited first, 6 of 6)

- [Training-Free Pretrained Model Merging](https://mlsanity.com/paper/2403.01753/) (2024)
- [Rumor Detection with a novel graph neural network approach](https://mlsanity.com/paper/2403.16206/) (2024)
- [Image Captioning in news report scenario](https://mlsanity.com/paper/2403.16209/) (2024)
- [Powerformer: A Section-adaptive Transformer for Power Flow Adjustment](https://mlsanity.com/paper/2401.02771/) (2024)
- [KA2ER: Knowledge Adaptive Amalgamation of ExpeRts for Medical Images Segmentation](https://mlsanity.com/paper/2410.21085/) (2024)
- [Uncertainty-Aware Multi-Expert Knowledge Distillation for Imbalanced Disease Grading](https://mlsanity.com/paper/2505.00592/) (2025)

## Abstract (arXiv)

Knowledge amalgamation (KA) aims to learn a compact student model to handle the joint objective from multiple teacher models that are are specialized for their own tasks respectively. Current methods focus on coarsely aligning teachers and students in the common representation space, making it difficult for the student to learn the proper decision boundaries from a set of heterogeneous teachers. Besides, the KL divergence in previous works only minimizes the probability distribution difference between teachers and the student, ignoring the intrinsic characteristics of teachers. Therefore, we propose a novel Contrastive Knowledge Amalgamation (CKA) framework, which introduces contrastive losses and an alignment loss to achieve intra-class cohesion and inter-class separation.Contrastive losses intra- and inter- models are designed to widen the distance between representations of different classes. The alignment loss is introduced to minimize the sample-level distribution differences of teacher-student models in the common representation space.Furthermore, the student learns heterogeneous unsupervised classification tasks through soft targets efficiently and flexibly in the task-level amalgamation. Extensive experiments on benchmarks demonstrate the generalization capability of CKA in the amalgamation of specific task as well as multiple tasks. Comprehensive ablation studies provide a further insight into our CKA.