# Knowledge Transfer via Dense Cross-Layer Mutual-Distillation

arXiv: https://arxiv.org/abs/2008.07816 · HTML: https://mlsanity.com/paper/2008.07816/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2020.

## Built on by (most cited first, 30 of 32)

- [EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation](https://mlsanity.com/paper/2111.12341/) (2021)
- [NORM: Knowledge Distillation via N-to-One Representation Matching](https://mlsanity.com/paper/2305.13803/) (2023)
- [Mutual Contrastive Learning for Visual Representation Learning](https://mlsanity.com/paper/2104.12565/) (2021)
- [A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation](https://mlsanity.com/paper/2307.12574/) (2023)
- [Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translation](https://mlsanity.com/paper/2109.01801/) (2021)
- [Learning Scene Structure Guidance via Cross-Task Knowledge Transfer for Single Depth Super-Resolution](https://mlsanity.com/paper/2103.12955/) (2021)
- [Spot-adaptive Knowledge Distillation](https://mlsanity.com/paper/2205.02399/) (2022)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All](https://mlsanity.com/paper/2405.16108/) (2024)
- [MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation](https://mlsanity.com/paper/2211.06018/) (2022)
- [Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation](https://mlsanity.com/paper/2308.09105/) (2023)
- [Dual-Tuning: Joint Prototype Transfer and Structure Regularization for Compatible Feature Learning](https://mlsanity.com/paper/2108.02959/) (2021)
- [Semi-Online Knowledge Distillation](https://mlsanity.com/paper/2111.11747/) (2021)
- [Bootstrapping ViTs: Towards Liberating Vision Transformers from Pre-training](https://mlsanity.com/paper/2112.03552/) (2021)
- [Learning From Biased Soft Labels](https://mlsanity.com/paper/2302.08155/) (2023)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [Self-Contrastive Learning: Single-viewed Supervised Contrastive Framework using Sub-network](https://mlsanity.com/paper/2106.15499/) (2021)
- [Scale Decoupled Distillation](https://mlsanity.com/paper/2403.13512/) (2024)
- [Enabling All In-Edge Deep Learning: A Literature Review](https://mlsanity.com/paper/2204.03326/) (2022)
- [Collaborative Teacher-Student Learning via Multiple Knowledge Transfer](https://mlsanity.com/paper/2101.08471/) (2021)
- [Improved Knowledge Distillation via Adversarial Collaboration](https://mlsanity.com/paper/2111.14356/) (2021)
- [Knowledge Distillation with Deep Supervision](https://mlsanity.com/paper/2202.07846/) (2022)
- [MLink: Linking Black-Box Models from Multiple Domains for Collaborative Inference](https://mlsanity.com/paper/2209.13883/) (2022)
- [Rethinking the Misalignment Problem in Dense Object Detection](https://mlsanity.com/paper/2108.12176/) (2021)
- [Triplet Knowledge Distillation](https://mlsanity.com/paper/2305.15975/) (2023)
- [Collaborative Knowledge Distillation via a Learning-by-Education Node Community](https://mlsanity.com/paper/2410.00074/) (2024)
- [Siamese Labels Auxiliary Learning](https://mlsanity.com/paper/2103.00200/) (2021)
- [Distilling Knowledge via Intermediate Classifiers](https://mlsanity.com/paper/2103.00497/) (2021)
- [Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation](https://mlsanity.com/paper/2110.00329/) (2021)
- [360^circ High-Resolution Depth Estimation via Uncertainty-aware Structural Knowledge Transfer](https://mlsanity.com/paper/2304.07967/) (2023)

## Abstract (arXiv)

Knowledge Distillation (KD) based methods adopt the one-way Knowledge Transfer (KT) scheme in which training a lower-capacity student network is guided by a pre-trained high-capacity teacher network. Recently, Deep Mutual Learning (DML) presented a two-way KT strategy, showing that the student network can be also helpful to improve the teacher network. In this paper, we propose Dense Cross-layer Mutual-distillation (DCM), an improved two-way KT method in which the teacher and student networks are trained collaboratively from scratch. To augment knowledge representation learning, well-designed auxiliary classifiers are added to certain hidden layers of both teacher and student networks. To boost KT performance, we introduce dense bidirectional KD operations between the layers appended with classifiers. After training, all auxiliary classifiers are discarded, and thus there are no extra parameters introduced to final models. We test our method on a variety of KT tasks, showing its superiorities over related methods. Code is available at https://github.com/sundw2014/DCM