# A Comprehensive Review on Deep Supervision: Theories and Applications

arXiv: https://arxiv.org/abs/2207.02376 · HTML: https://mlsanity.com/paper/2207.02376/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 119 references that are in the map.
- First posted on arXiv 6 Jul 2022.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)
- [Deeply-Recursive Convolutional Network for Image Super-Resolution](https://mlsanity.com/paper/1511.04491/) (2015)
- [Component Divide-and-Conquer for Real-World Image Super-Resolution](https://mlsanity.com/paper/2008.01928/) (2020)
- [Deeply supervised salient object detection with short connections](https://mlsanity.com/paper/1611.04849/) (2016)
- [Learning Deep Models for Face Anti-Spoofing: Binary or Auxiliary Supervision](https://mlsanity.com/paper/1803.11097/) (2018)
- [Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks](https://mlsanity.com/paper/1710.01992/) (2017)
- [Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation](https://mlsanity.com/paper/1705.08302/) (2017)
- [3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes](https://mlsanity.com/paper/1607.00582/) (2016)
- [DSOD: Learning Deeply Supervised Object Detectors from Scratch](https://mlsanity.com/paper/1708.01241/) (2017)
- [Multi-Scale Structure-Aware Network for Human Pose Estimation](https://mlsanity.com/paper/1803.09894/) (2018)
- [Training Deeper Convolutional Networks with Deep Supervision](https://mlsanity.com/paper/1505.02496/) (2015)
- [SNE-RoadSeg+: Rethinking Depth-Normal Translation and Deep Supervision for Freespace Detection](https://mlsanity.com/paper/2107.14599/) (2021)
- [Object Detection from Scratch with Deep Supervision](https://mlsanity.com/paper/1809.09294/) (2018)
- [Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing](https://mlsanity.com/paper/1612.02699/) (2016)
- [Deep Supervision with Intermediate Concepts](https://mlsanity.com/paper/1801.03399/) (2018)
- [LabelEnc: A New Intermediate Supervision Method for Object Detection](https://mlsanity.com/paper/2007.03282/) (2020)
- [DNA: Deeply-supervised Nonlinear Aggregation for Salient Object Detection](https://mlsanity.com/paper/1903.12476/) (2019)
- [Semantic Edge Detection with Diverse Deep Supervision](https://mlsanity.com/paper/1804.02864/) (2018)
- [HED-UNet: Combined Segmentation and Edge Detection for Monitoring the Antarctic Coastline](https://mlsanity.com/paper/2103.01849/) (2021)
- [A Multi-stage Framework with Context Information Fusion Structure for Skin Lesion Segmentation](https://mlsanity.com/paper/1810.07075/) (2018)
- [Deep Supervision for Pancreatic Cyst Segmentation in Abdominal CT Scans](https://mlsanity.com/paper/1706.07346/) (2017)

## Built on by (most cited first, 10 of 10)

- [DeepMIM: Deep Supervision for Masked Image Modeling](https://mlsanity.com/paper/2303.08817/) (2023)
- [Solutions to Elliptic and Parabolic Problems via Finite Difference Based Unsupervised Small Linear Convolutional Neural Networks](https://mlsanity.com/paper/2311.00259/) (2023)
- [Verification of the Implicit World Model in a Generative Model via Adversarial Sequences](https://mlsanity.com/paper/2602.05903/) (2026)
- [Joint Learning for Scattered Point Cloud Understanding with Hierarchical Self-Distillation](https://mlsanity.com/paper/2312.16902/) (2023)
- [EU-Nets: Enhanced, Explainable and Parsimonious U-Nets](https://mlsanity.com/paper/2502.18122/) (2025)
- [Multi-output Deep-Supervised Classifier Chains for Plant Pathology](https://mlsanity.com/paper/2507.20125/) (2025)
- [A Semantic Segmentation Algorithm for Pleural Effusion Based on DBIF-AUNet](https://mlsanity.com/paper/2508.06191/) (2025)
- [LION-DG: Layer-Informed Initialization with Deep Gradient Protocols for Accelerated Neural Network Training](https://mlsanity.com/paper/2601.02105/) (2026)
- [OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization](https://mlsanity.com/paper/2605.20276/) (2026)
- [From Gradient Clipping to Structural Refinement: Improving DPSGD for Medical Image Segmentation](https://mlsanity.com/paper/2606.21763/) (2026)

## Abstract (arXiv)

Deep supervision, or known as 'intermediate supervision' or 'auxiliary supervision', is to add supervision at hidden layers of a neural network. This technique has been increasingly applied in deep neural network learning systems for various computer vision applications recently. There is a consensus that deep supervision helps improve neural network performance by alleviating the gradient vanishing problem, as one of the many strengths of deep supervision. Besides, in different computer vision applications, deep supervision can be applied in different ways. How to make the most use of deep supervision to improve network performance in different applications has not been thoroughly investigated. In this paper, we provide a comprehensive in-depth review of deep supervision in both theories and applications. We propose a new classification of different deep supervision networks, and discuss advantages and limitations of current deep supervision networks in computer vision applications.