# Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning

arXiv: https://arxiv.org/abs/1906.10546 · HTML: https://mlsanity.com/paper/1906.10546/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2019.

## Built on by (most cited first, 28 of 28)

- [One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation](https://mlsanity.com/paper/2310.19444/) (2023)
- [Factorizing Knowledge in Neural Networks](https://mlsanity.com/paper/2207.03337/) (2022)
- [Training-Free Pretrained Model Merging](https://mlsanity.com/paper/2403.01753/) (2024)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model](https://mlsanity.com/paper/2310.17653/) (2023)
- [Class-Incremental Learning via Knowledge Amalgamation](https://mlsanity.com/paper/2209.02112/) (2022)
- [Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning](https://mlsanity.com/paper/2410.14633/) (2024)
- [Contrastive Knowledge Amalgamation for Unsupervised Image Classification](https://mlsanity.com/paper/2307.14781/) (2023)
- [Training-free Heterogeneous Model Merging](https://mlsanity.com/paper/2501.00061/) (2024)
- [Knowledge Amalgamation for Object Detection with Transformers](https://mlsanity.com/paper/2203.03187/) (2022)
- [Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures](https://mlsanity.com/paper/2405.18524/) (2024)
- [Cross-Architecture Auxiliary Feature Space Translation for Efficient Few-Shot Personalized Object Detection](https://mlsanity.com/paper/2407.01193/) (2024)
- [HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models](https://mlsanity.com/paper/2506.19072/) (2025)
- [Towards Communication-Efficient and Privacy-Preserving Federated Representation Learning](https://mlsanity.com/paper/2109.14611/) (2021)
- [Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Models](https://mlsanity.com/paper/2112.07327/) (2021)
- [Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal](https://mlsanity.com/paper/2305.00441/) (2023)
- [MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment](https://mlsanity.com/paper/2509.14001/) (2025)
- [Optimizer Amalgamation](https://mlsanity.com/paper/2203.06474/) (2022)
- [Federated Selective Aggregation for Knowledge Amalgamation](https://mlsanity.com/paper/2207.13309/) (2022)
- [From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models](https://mlsanity.com/paper/2210.05230/) (2022)
- [Knowledge Distillation on Spatial-Temporal Graph Convolutional Network for Traffic Prediction](https://mlsanity.com/paper/2401.11798/) (2024)
- [Self-Regulated Data-Free Knowledge Amalgamation for Text Classification](https://mlsanity.com/paper/2406.15476/) (2024)
- [DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging](https://mlsanity.com/paper/2508.01148/) (2025)
- [Anno-incomplete Multi-dataset Detection](https://mlsanity.com/paper/2408.16247/) (2024)
- [Transferring Social Network Knowledge from Multiple GNN Teachers to Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2508.06663/) (2025)
- [UNIFORM: Unifying Knowledge from Large-scale and Diverse Pre-trained Models](https://mlsanity.com/paper/2508.19498/) (2025)
- [Simplifying Knowledge Transfer in Pretrained Models](https://mlsanity.com/paper/2510.22208/) (2025)
- [Distillation for Efficient Multitask Manipulation Policies via Conditional Flow Matching](https://mlsanity.com/paper/2609.28107/) (2026)

## Abstract (arXiv)

An increasing number of well-trained deep networks have been released online by researchers and developers, enabling the community to reuse them in a plug-and-play way without accessing the training annotations. However, due to the large number of network variants, such public-available trained models are often of different architectures, each of which being tailored for a specific task or dataset. In this paper, we study a deep-model reusing task, where we are given as input pre-trained networks of heterogeneous architectures specializing in distinct tasks, as teacher models. We aim to learn a multitalented and light-weight student model that is able to grasp the integrated knowledge from all such heterogeneous-structure teachers, again without accessing any human annotation. To this end, we propose a common feature learning scheme, in which the features of all teachers are transformed into a common space and the student is enforced to imitate them all so as to amalgamate the intact knowledge. We test the proposed approach on a list of benchmarks and demonstrate that the learned student is able to achieve very promising performance, superior to those of the teachers in their specialized tasks.