mlsanityData to 14 Oct 2026

Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning

arXiv 2306.06634 ↗First posted 11 Jun 2023cs.CVcited by 16

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Multi-Teacher knowledge distillation provides students with additional supervision from multiple pre-trained teachers with diverse information sources. Most existing methods explore different weighting strategies to obtain a powerful ensemble teacher, while ignoring the student with poor learning ability may not benefit from such specialized integrated knowledge. To address this problem, we propose Adaptive Multi-teacher Knowledge Distillation with Meta-Learning (MMKD) to supervise student with appropriate knowledge from a tailored ensemble teacher. With the help of a meta-weight network, the diverse yet compatible teacher knowledge in the output layer and intermediate layers is jointly leveraged to enhance the student performance. Extensive experiments on multiple benchmark datasets validate the effectiveness and flexibility of our methods. Code is available: https://github.com/Rorozhl/MMKD.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 19 of its 25 references that are in the map. First posted on arXiv 11 Jun 2023.

Builds on

19 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Knowledge Distillation with the Reused Teacher Classifier26 Mar
  2. 2021Confidence-Aware Multi-Teacher Knowledge Distillation30 Dec

Other references · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  9. 2019Contrastive Representation Distillationcited by 748
  10. 2019Similarity-Preserving Knowledge Distillationcited by 516
  11. 2019Variational Information Distillation for Knowledge Transfercited by 324
  12. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
5 more
  1. 2020Cross-Layer Distillation with Semantic Calibrationcited by 143
  2. 2019Online Knowledge Distillation with Diverse Peerscited by 128
  3. 2020Reinforced Multi-Teacher Selection for Knowledge Distillationcited by 63
  4. 2019Learning What and Where to Transfercited by 43
  5. 2023A Geometric Perspective on Diffusion Modelscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognitioncited by 16
  2. 2024TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaborationcited by 5
  3. 2025Ice-FMBench: A Foundation Model Benchmark for Sea Ice Type Segmentationcited by 5
  4. 2025DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainercited by 4
  5. 2025Learning Task-Agnostic Representations through Multi-Teacher Distillationcited by 4
  6. 2026Multi-Teacher Ensemble Distillation: A Mathematical Framework for Probability-Domain Knowledge Aggregationcited by 3
  7. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  8. 2025DCSNet: A Lightweight Knowledge Distillation-Based Model with Explainable AI for Lung Cancer Diagnosis from Histopathological Imagescited by 1
  9. 2025Distilled-3DGS:Distilled 3D Gaussian Splattingcited by 1
  10. 2025Knowledge Distillation of Uncertainty using Deep Latent Factor Modelcited by 1
  11. 2026When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecastingcited by 1
  12. 2025Beyond Real Weights: Hypercomplex Representations for Stable Quantizationcited by 0
4 more
  1. 2026Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancementcited by 0
  2. 2026Evolving Knowledge Distillation for Lightweight Neural Machine Translationcited by 0
  3. 2026Dynamic Influence-Weighted Distillation for Single-IMU Activity Recognitioncited by 0
  4. 2026Adaptive Mutual Distillation for Balanced Multi-Task Post-Training of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillation
  2. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learning
  3. 2022Knowledge Distillation via Weighted Ensemble of Teaching Assistants
  4. 2026Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Framework
  5. 2025Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity
  6. 2026Adaptive Weighting in Knowledge Distillation: An Axiomatic Framework for Multi-Scale Teacher Ensemble Optimization
  7. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillation
  8. 2022Learn From the Past: Experience Ensemble Knowledge Distillation