# Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning

arXiv: https://arxiv.org/abs/2306.06634 · HTML: https://mlsanity.com/paper/2306.06634/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Forward and Reverse Gradient-Based Hyperparameter Optimization](https://mlsanity.com/paper/1703.01785/) (2017)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022)
- [Online Knowledge Distillation with Diverse Peers](https://mlsanity.com/paper/1912.00350/) (2019)
- [Reinforced Multi-Teacher Selection for Knowledge Distillation](https://mlsanity.com/paper/2012.06048/) (2020)
- [Learning What and Where to Transfer](https://mlsanity.com/paper/1905.05901/) (2019)
- [Confidence-Aware Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2201.00007/) (2021)
- [A Geometric Perspective on Diffusion Models](https://mlsanity.com/paper/2305.19947/) (2023)

## Built on by (most cited first, 16 of 16)

- [Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition](https://mlsanity.com/paper/2502.18510/) (2025)
- [TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration](https://mlsanity.com/paper/2410.12183/) (2024)
- [Ice-FMBench: A Foundation Model Benchmark for Sea Ice Type Segmentation](https://mlsanity.com/paper/2503.22516/) (2025)
- [DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer](https://mlsanity.com/paper/2505.15133/) (2025)
- [Learning Task-Agnostic Representations through Multi-Teacher Distillation](https://mlsanity.com/paper/2510.18680/) (2025)
- [Multi-Teacher Ensemble Distillation: A Mathematical Framework for Probability-Domain Knowledge Aggregation](https://mlsanity.com/paper/2601.09165/) (2026)
- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [DCSNet: A Lightweight Knowledge Distillation-Based Model with Explainable AI for Lung Cancer Diagnosis from Histopathological Images](https://mlsanity.com/paper/2505.09334/) (2025)
- [Distilled-3DGS:Distilled 3D Gaussian Splatting](https://mlsanity.com/paper/2508.14037/) (2025)
- [Knowledge Distillation of Uncertainty using Deep Latent Factor Model](https://mlsanity.com/paper/2510.19290/) (2025)
- [When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting](https://mlsanity.com/paper/2606.19363/) (2026)
- [Beyond Real Weights: Hypercomplex Representations for Stable Quantization](https://mlsanity.com/paper/2512.08524/) (2025)
- [Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement](https://mlsanity.com/paper/2603.24208/) (2026)
- [Evolving Knowledge Distillation for Lightweight Neural Machine Translation](https://mlsanity.com/paper/2605.09924/) (2026)
- [Dynamic Influence-Weighted Distillation for Single-IMU Activity Recognition](https://mlsanity.com/paper/2608.24904/) (2026)
- [Adaptive Mutual Distillation for Balanced Multi-Task Post-Training of Large Language Models](https://mlsanity.com/paper/2610.02856/) (2026)

## Abstract (arXiv)

Multi-Teacher knowledge distillation provides students with additional supervision from multiple pre-trained teachers with diverse information sources. Most existing methods explore different weighting strategies to obtain a powerful ensemble teacher, while ignoring the student with poor learning ability may not benefit from such specialized integrated knowledge. To address this problem, we propose Adaptive Multi-teacher Knowledge Distillation with Meta-Learning (MMKD) to supervise student with appropriate knowledge from a tailored ensemble teacher. With the help of a meta-weight network, the diverse yet compatible teacher knowledge in the output layer and intermediate layers is jointly leveraged to enhance the student performance. Extensive experiments on multiple benchmark datasets validate the effectiveness and flexibility of our methods. Code is available: https://github.com/Rorozhl/MMKD.