# MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution arXiv: https://arxiv.org/abs/2404.09571 ยท HTML: https://mlsanity.com/paper/2404.09571/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 47 of its 77 references that are in the map. - First posted on arXiv 15 Apr 2024. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 30 of 47) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021) - [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016) - [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021) - [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017) - [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021) - [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021) - [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018) - [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014) - [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016) - [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019) - [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016) - [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015) - [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019) - [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021) - [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021) - [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023) - [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018) - [Transformer for Single Image Super-Resolution](https://mlsanity.com/paper/2108.11084/) (2021) - [Interpreting Super-Resolution Networks with Local Attribution Maps](https://mlsanity.com/paper/2011.11036/) (2020) - [Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration](https://mlsanity.com/paper/2209.11345/) (2022) - [Knowledge Distillation via the Target-aware Transformer](https://mlsanity.com/paper/2205.10793/) (2022) - [SwinFIR: Revisiting the SwinIR with Fast Fourier Convolution and Improved Training for Image Super-Resolution](https://mlsanity.com/paper/2208.11247/) (2022) - [HAT: Hybrid Attention Transformer for Image Restoration](https://mlsanity.com/paper/2309.05239/) (2023) - [Reinforced Multi-Teacher Selection for Knowledge Distillation](https://mlsanity.com/paper/2012.06048/) (2020) - [Transformers in Remote Sensing: A Survey](https://mlsanity.com/paper/2209.01206/) (2022) - [Learning with Privileged Information for Efficient Image Super-Resolution](https://mlsanity.com/paper/2007.07524/) (2020) ## Built on by (most cited first, 17 of 17) - [The Tenth NTIRE 2025 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2504.10686/) (2025) - [HIIF: Hierarchical Encoding based Implicit Image Function for Continuous Super-resolution](https://mlsanity.com/paper/2412.03748/) (2024) - [AIM 2024 Challenge on Efficient Video Super-Resolution for AV1 Compressed Content](https://mlsanity.com/paper/2409.17256/) (2024) - [C2D-ISR: Optimizing Attention-based Image Super-resolution from Continuous to Discrete Scales](https://mlsanity.com/paper/2503.13740/) (2025) - [RTSR: A Real-Time Super-Resolution Model for AV1 Compressed Content](https://mlsanity.com/paper/2411.13362/) (2024) - [DMM: Building a Versatile Image Generation Model via Distillation-Based Model Merging](https://mlsanity.com/paper/2504.12364/) (2025) - [Blind Video Super-Resolution based on Implicit Kernels](https://mlsanity.com/paper/2503.07856/) (2025) - [Empowering Source-Free Domain Adaptation via MLLM-Guided Reliability-Based Curriculum Learning](https://mlsanity.com/paper/2405.18376/) (2024) - [Agglomerating Large Vision Encoders via Distillation for VFSS Segmentation](https://mlsanity.com/paper/2504.02351/) (2025) - [GFix: Perceptually Enhanced Gaussian Splatting Video Compression](https://mlsanity.com/paper/2511.06953/) (2025) - [Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking](https://mlsanity.com/paper/2412.20002/) (2024) - [Compressed Video Super-Resolution based on Hierarchical Encoding](https://mlsanity.com/paper/2506.14381/) (2025) - [Collaborative Unlabeled Data Optimization](https://mlsanity.com/paper/2505.14117/) (2025) - [CAST: Contrastive Adaptation and Distillation for Semi-Supervised Instance Segmentation](https://mlsanity.com/paper/2505.21904/) (2025) - [Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement](https://mlsanity.com/paper/2603.24208/) (2026) - [Training a Student Expert via Semi-Supervised Foundation Model Distillation](https://mlsanity.com/paper/2604.03841/) (2026) - [DTKDP: A Dual Teacher Knowledge Distillation and Pruning Framework for Lightweight Oriented SAR Ship Detection](https://mlsanity.com/paper/2609.24872/) (2026) ## Abstract (arXiv) Knowledge distillation (KD) has emerged as a promising technique in deep learning, typically employed to enhance a compact student network through learning from their high-performance but more complex teacher variant. When applied in the context of image super-resolution, most KD approaches are modified versions of methods developed for other computer vision tasks, which are based on training strategies with a single teacher and simple loss functions. In this paper, we propose a novel Multi-Teacher Knowledge Distillation (MTKD) framework specifically for image super-resolution. It exploits the advantages of multiple teachers by combining and enhancing the outputs of these teacher models, which then guides the learning process of the compact student network. To achieve more effective learning performance, we have also developed a new wavelet-based loss function for MTKD, which can better optimize the training process by observing differences in both the spatial and frequency domains. We fully evaluate the effectiveness of the proposed method by comparing it to five commonly used KD methods for image super-resolution based on three popular network architectures. The results show that the proposed MTKD method achieves evident improvements in super-resolution performance, up to 0.46dB (based on PSNR), over state-of-the-art KD approaches across different network structures. The source code of MTKD will be made available here for public evaluation.