# Knowledge Flow: Improve Upon Your Teachers

arXiv: https://arxiv.org/abs/1904.05878 · HTML: https://mlsanity.com/paper/1904.05878/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 30 of 36)

- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities](https://mlsanity.com/paper/2211.06679/) (2022)
- [Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights](https://mlsanity.com/paper/2209.14733/) (2022)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Complementary Relation Contrastive Distillation](https://mlsanity.com/paper/2103.16367/) (2021)
- [Model Zoos: A Dataset of Diverse Populations of Neural Network Models](https://mlsanity.com/paper/2209.14764/) (2022)
- [Zoo-Tuning: Adaptive Transfer from a Zoo of Models](https://mlsanity.com/paper/2106.15434/) (2021)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model](https://mlsanity.com/paper/2310.17653/) (2023)
- [Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.03319/) (2021)
- [Distilling Efficient Vision Transformers from CNNs for Semantic Segmentation](https://mlsanity.com/paper/2310.07265/) (2023)
- [Boosting Light-Weight Depth Estimation Via Knowledge Distillation](https://mlsanity.com/paper/2105.06143/) (2021)
- [All-in-One: Transferring Vision Foundation Models into Stereo Matching](https://mlsanity.com/paper/2412.09912/) (2024)
- [ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization](https://mlsanity.com/paper/2210.09236/) (2022)
- [KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks](https://mlsanity.com/paper/2110.02432/) (2021)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [Federated Learning with Privacy-Preserving Ensemble Attention Distillation](https://mlsanity.com/paper/2210.08464/) (2022)
- [Representation Consolidation for Training Expert Students](https://mlsanity.com/paper/2107.08039/) (2021)
- [Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/2205.14410/) (2022)
- [Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models](https://mlsanity.com/paper/2206.03726/) (2022)
- [T-UDA: Temporal Unsupervised Domain Adaptation in Sequential Point Clouds](https://mlsanity.com/paper/2309.08302/) (2023)
- [Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation](https://mlsanity.com/paper/2110.09674/) (2021)
- [Dense Depth Distillation with Out-of-Distribution Simulated Images](https://mlsanity.com/paper/2208.12464/) (2022)
- [Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models](https://mlsanity.com/paper/2304.13718/) (2023)
- [Efficient Stitchable Task Adaptation](https://mlsanity.com/paper/2311.17352/) (2023)
- [Teachers in concordance for pseudo-labeling of 3D sequential data](https://mlsanity.com/paper/2207.06079/) (2022)
- [A Model Zoo of Vision Transformers](https://mlsanity.com/paper/2504.10231/) (2025)
- [A Model Zoo on Phase Transitions in Neural Networks](https://mlsanity.com/paper/2504.18072/) (2025)

## Abstract (arXiv)

A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves 'knowledge' from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other 'knowledge exchange' methods.