# Unifying Heterogeneous Classifiers with Distillation

arXiv: https://arxiv.org/abs/1904.06062 · HTML: https://mlsanity.com/paper/1904.06062/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2019.

## Built on by (most cited first, 25 of 25)

- [Adaptive Multi-Teacher Multi-level Knowledge Distillation](https://mlsanity.com/paper/2103.04062/) (2021)
- [Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks](https://mlsanity.com/paper/2210.16103/) (2022)
- [Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation](https://mlsanity.com/paper/2107.12087/) (2021)
- [Cosine Model Watermarking Against Ensemble Distillation](https://mlsanity.com/paper/2203.02777/) (2022)
- [Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation](https://mlsanity.com/paper/2308.09105/) (2023)
- [FedDKD: Federated Learning with Decentralized Knowledge Distillation](https://mlsanity.com/paper/2205.00706/) (2022)
- [Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning](https://mlsanity.com/paper/2410.14633/) (2024)
- [Contrastive Knowledge Amalgamation for Unsupervised Image Classification](https://mlsanity.com/paper/2307.14781/) (2023)
- [Representation Consolidation for Training Expert Students](https://mlsanity.com/paper/2107.08039/) (2021)
- [Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models](https://mlsanity.com/paper/2201.00558/) (2022)
- [Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks](https://mlsanity.com/paper/2410.01483/) (2024)
- [Revisiting Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2105.10633/) (2021)
- [Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Models](https://mlsanity.com/paper/2112.07327/) (2021)
- [Heterogeneous Generative Knowledge Distillation with Masked Image Modeling](https://mlsanity.com/paper/2309.09571/) (2023)
- [Pool of Experts: Realtime Querying Specialized Knowledge in Massive Neural Networks](https://mlsanity.com/paper/2107.01354/) (2021)
- [Distillation of Human-Object Interaction Contexts for Action Recognition](https://mlsanity.com/paper/2112.09448/) (2021)
- [From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models](https://mlsanity.com/paper/2210.05230/) (2022)
- [Swing Distillation: A Privacy-Preserving Knowledge Distillation Framework](https://mlsanity.com/paper/2212.08349/) (2022)
- [Self-Regulated Data-Free Knowledge Amalgamation for Text Classification](https://mlsanity.com/paper/2406.15476/) (2024)
- [3M-Health: Multimodal Multi-Teacher Knowledge Distillation for Mental Health Detection](https://mlsanity.com/paper/2407.09020/) (2024)
- [Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph](https://mlsanity.com/paper/2109.04400/) (2021)
- [Federated One-Shot Learning with Data Privacy and Objective-Hiding](https://mlsanity.com/paper/2504.21182/) (2025)
- [Expandable Residual Approximation for Knowledge Distillation](https://mlsanity.com/paper/2508.16050/) (2025)
- [Adaptive Dual-Mode Distillation with Incentive Schemes for Scalable, Heterogeneous Federated Learning on Non-IID Data](https://mlsanity.com/paper/2509.22507/) (2025)
- [Functional Properties of the Focal-Entropy](https://mlsanity.com/paper/2603.02533/) (2026)

## Abstract (arXiv)

In this paper, we study the problem of unifying knowledge from a set of classifiers with different architectures and target classes into a single classifier, given only a generic set of unlabelled data. We call this problem Unifying Heterogeneous Classifiers (UHC). This problem is motivated by scenarios where data is collected from multiple sources, but the sources cannot share their data, e.g., due to privacy concerns, and only privately trained models can be shared. In addition, each source may not be able to gather data to train all classes due to data availability at each source, and may not be able to train the same classification model due to different computational resources. To tackle this problem, we propose a generalisation of knowledge distillation to merge HCs. We derive a probabilistic relation between the outputs of HCs and the probability over all classes. Based on this relation, we propose two classes of methods based on cross-entropy minimisation and matrix factorisation, which allow us to estimate soft labels over all classes from unlabelled samples and use them in lieu of ground truth labels to train a unified classifier. Our extensive experiments on ImageNet, LSUN, and Places365 datasets show that our approaches significantly outperform a naive extension of distillation and can achieve almost the same accuracy as classifiers that are trained in a centralised, supervised manner.