mlsanityData to 14 Oct 2026

Knowledge Flow: Improve Upon Your Teachers

arXiv 1904.05878 ↗First posted 11 Apr 2019cs.LG stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves 'knowledge' from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other 'knowledge exchange' methods.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 10; 2023: 10; 2024: 1; 2025: 3.

36 papers.

Sort:

By area:

  1. 2023Deep Model Fusion: A Surveycited by 91
  2. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  3. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 85
  4. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 64
  5. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  6. 2021A Survey on Green Deep Learningcited by 52
  7. 2021Complementary Relation Contrastive Distillationcited by 42
  8. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 39
  9. 2021Zoo-Tuning: Adaptive Transfer from a Zoo of Modelscited by 35
  10. 2021Transferability Metrics for Selecting Source Model Ensemblescited by 24
  11. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  12. 2021Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learningcited by 18
24 more
  1. 2023Distilling Efficient Vision Transformers from CNNs for Semantic Segmentationcited by 18
  2. 2021Boosting Light-Weight Depth Estimation Via Knowledge Distillationcited by 16
  3. 2024All-in-One: Transferring Vision Foundation Models into Stereo Matchingcited by 16
  4. 2022ZooD: Exploiting Model Zoo for Out-of-Distribution Generalizationcited by 12
  5. 2021KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Taskscited by 8
  6. 2021Compacting Deep Neural Networks for Internet of Things: Methods and Applicationscited by 6
  7. 2022Federated Learning with Privacy-Preserving Ensemble Attention Distillationcited by 6
  8. 2021Representation Consolidation for Training Expert Studentscited by 5
  9. 2022Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learningcited by 5
  10. 2022Hub-Pathway: Transfer Learning from A Hub of Pre-trained Modelscited by 5
  11. 2023T-UDA: Temporal Unsupervised Domain Adaptation in Sequential Point Cloudscited by 5
  12. 2021Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillationcited by 4
  13. 2022Dense Depth Distillation with Out-of-Distribution Simulated Imagescited by 4
  14. 2023Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Modelscited by 4
  15. 2023Efficient Stitchable Task Adaptationcited by 4
  16. 2022Teachers in concordance for pseudo-labeling of 3D sequential datacited by 3
  17. 2025A Model Zoo of Vision Transformerscited by 3
  18. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  19. 2023Topology-preserving Adversarial Training for Alleviating Natural Accuracy Degradationcited by 1
  20. 2022Learning to Advise and Learning from Advice in Cooperative Multi-Agent Reinforcement Learningcited by 0
  21. 2023Smaller3d: Smaller Models for 3D Semantic Segmentation Using Minkowski Engine and Knowledge Distillation Methodscited by 0
  22. 2023Model-Based Reinforcement Learning with Multi-Task Offline Pretrainingcited by 0
  23. 2023RRR-Net: Reusing, Reducing, and Recycling a Deep Backbone Networkcited by 0
  24. 2025SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillationcited by 0