# Deeply-supervised Knowledge Synergy

arXiv: https://arxiv.org/abs/1906.00675 · HTML: https://mlsanity.com/paper/1906.00675/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2019.

## Built on by (most cited first, 18 of 18)

- [Self-Supervised Monocular Depth and Ego-Motion Estimation in Endoscopy: Appearance Flow to the Rescue](https://mlsanity.com/paper/2112.08122/) (2021)
- [From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels](https://mlsanity.com/paper/2303.13005/) (2023)
- [MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition](https://mlsanity.com/paper/2208.05768/) (2022)
- [Contrastive Deep Supervision](https://mlsanity.com/paper/2207.05306/) (2022)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [URCDC-Depth: Uncertainty Rectified Cross-Distillation with CutFlip for Monocular Depth Estimation](https://mlsanity.com/paper/2302.08149/) (2023)
- [MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation](https://mlsanity.com/paper/2211.06018/) (2022)
- [Weakly Supervised Monocular 3D Detection with a Single-View Image](https://mlsanity.com/paper/2402.19144/) (2024)
- [LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval](https://mlsanity.com/paper/2203.05465/) (2022)
- [Self-Distillation for Further Pre-training of Transformers](https://mlsanity.com/paper/2210.02871/) (2022)
- [Network Pruning via Resource Reallocation](https://mlsanity.com/paper/2103.01847/) (2021)
- [Knowledge Distillation with Deep Supervision](https://mlsanity.com/paper/2202.07846/) (2022)
- [Deeply Supervised Layer Selective Attention Network: Towards Label-Efficient Learning for Medical Image Classification](https://mlsanity.com/paper/2209.13844/) (2022)
- [RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation](https://mlsanity.com/paper/2301.08092/) (2023)
- [SAEN-BGS: Energy-Efficient Spiking AutoEncoder Network for Background Subtraction](https://mlsanity.com/paper/2505.07336/) (2025)
- [Two-Step Image Dehazing with Intra-domain and Inter-domain Adaptation](https://mlsanity.com/paper/2102.03501/) (2021)
- [Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation](https://mlsanity.com/paper/2110.00329/) (2021)
- [Extracting knowledge from features with multilevel abstraction](https://mlsanity.com/paper/2112.13642/) (2021)

## Abstract (arXiv)

Convolutional Neural Networks (CNNs) have become deeper and more complicated compared with the pioneering AlexNet. However, current prevailing training scheme follows the previous way of adding supervision to the last layer of the network only and propagating error information up layer-by-layer. In this paper, we propose Deeply-supervised Knowledge Synergy (DKS), a new method aiming to train CNNs with improved generalization ability for image classification tasks without introducing extra computational cost during inference. Inspired by the deeply-supervised learning scheme, we first append auxiliary supervision branches on top of certain intermediate network layers. While properly using auxiliary supervision can improve model accuracy to some degree, we go one step further to explore the possibility of utilizing the probabilistic knowledge dynamically learnt by the classifiers connected to the backbone network as a new regularization to improve the training. A novel synergy loss, which considers pairwise knowledge matching among all supervision branches, is presented. Intriguingly, it enables dense pairwise knowledge matching operations in both top-down and bottom-up directions at each training iteration, resembling a dynamic synergy process for the same task. We evaluate DKS on image classification datasets using state-of-the-art CNN architectures, and show that the models trained with it are consistently better than the corresponding counterparts. For instance, on the ImageNet classification benchmark, our ResNet-152 model outperforms the baseline model with a 1.47% margin in Top-1 accuracy. Code is available at https://github.com/sundw2014/DKS.