# Training Deeper Convolutional Networks with Deep Supervision

arXiv: https://arxiv.org/abs/1505.02496 · HTML: https://mlsanity.com/paper/1505.02496/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2015.

## Built on by (most cited first, 30 of 36)

- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [Video-based Person Re-identification with Spatial and Temporal Memory Networks](https://mlsanity.com/paper/2108.09039/) (2021)
- [Contrastive Deep Supervision](https://mlsanity.com/paper/2207.05306/) (2022)
- [Medical Image Segmentation with 3D Convolutional Neural Networks: A Survey](https://mlsanity.com/paper/2108.08467/) (2021)
- [DeepMIM: Deep Supervision for Masked Image Modeling](https://mlsanity.com/paper/2303.08817/) (2023)
- [Guiding a Diffusion Transformer with the Internal Dynamics of Itself](https://mlsanity.com/paper/2512.24176/) (2025)
- [MaskRange: A Mask-classification Model for Range-view based LiDAR Segmentation](https://mlsanity.com/paper/2206.12073/) (2022)
- [A Comprehensive Review on Deep Supervision: Theories and Applications](https://mlsanity.com/paper/2207.02376/) (2022)
- [Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16235/) (2023)
- [PINQI: An End-to-End Physics-Informed Approach to Learned Quantitative MRI Reconstruction](https://mlsanity.com/paper/2306.11023/) (2023)
- [3DPX: Progressive 2D-to-3D Oral Image Reconstruction with Hybrid MLP-CNN Networks](https://mlsanity.com/paper/2408.01292/) (2024)
- [Understanding the Tricks of Deep Learning in Medical Image Segmentation: Challenges and Future Directions](https://mlsanity.com/paper/2209.10307/) (2022)
- [MIST: A Simple and Scalable End-To-End 3D Medical Imaging Segmentation Framework](https://mlsanity.com/paper/2407.21343/) (2024)
- [Deeply Supervised Skin Lesions Diagnosis with Stage and Branch Attention](https://mlsanity.com/paper/2205.04326/) (2022)
- [You Only Look Once at Anytime (AnytimeYOLO): Analysis and Optimization of Early-Exits for Object-Detection](https://mlsanity.com/paper/2503.17497/) (2025)
- [FreqU-FNet: Frequency-Aware U-Net for Imbalanced Medical Image Segmentation](https://mlsanity.com/paper/2505.17544/) (2025)
- [A Densely Interconnected Network for Deep Learning Accelerated MRI](https://mlsanity.com/paper/2207.02073/) (2022)
- [Progressive DeepSSM: Training Methodology for Image-To-Shape Deep Models](https://mlsanity.com/paper/2310.01529/) (2023)
- [MEDPSeg: Hierarchical polymorphic multitask learning for the segmentation of ground-glass opacities, consolidation, and pulmonary structures on computed tomography](https://mlsanity.com/paper/2312.02365/) (2023)
- [MALT: Multi-scale Action Learning Transformer for Online Action Detection](https://mlsanity.com/paper/2405.20892/) (2024)
- [Uncertainty, Edge, and Reverse-Attention Guided Generative Adversarial Network for Automatic Building Detection in Remotely Sensed Images](https://mlsanity.com/paper/2112.05335/) (2021)
- [DIGEST: Deeply supervIsed knowledGE tranSfer neTwork learning for brain tumor segmentation with incomplete multi-modal MRI scans](https://mlsanity.com/paper/2211.07993/) (2022)
- [VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformer](https://mlsanity.com/paper/2502.16654/) (2025)
- [TopoMortar: A dataset to evaluate image segmentation methods focused on topology accuracy](https://mlsanity.com/paper/2503.03365/) (2025)
- [A Multiple Classifier Approach for Concatenate-Designed Neural Networks](https://mlsanity.com/paper/2101.05457/) (2021)
- [Feature Noise Resilient for QoS Prediction with Probabilistic Deep Supervision](https://mlsanity.com/paper/2308.02580/) (2023)
- [ElasticLaneNet: An Efficient Geometry-Flexible Approach for Lane Detection](https://mlsanity.com/paper/2312.10389/) (2023)
- [BIMM: Brain Inspired Masked Modeling for Video Representation Learning](https://mlsanity.com/paper/2405.12757/) (2024)
- [Graph-Boosted Attentive Network for Semantic Body Parsing](https://mlsanity.com/paper/2407.05924/) (2024)
- [3DPX: Single Panoramic X-ray Analysis Guided by 3D Oral Structure Reconstruction](https://mlsanity.com/paper/2409.18701/) (2024)

## Abstract (arXiv)

One of the most promising ways of improving the performance of deep convolutional neural networks is by increasing the number of convolutional layers. However, adding layers makes training more difficult and computationally expensive. In order to train deeper networks, we propose to add auxiliary supervision branches after certain intermediate layers during training. We formulate a simple rule of thumb to determine where these branches should be added. The resulting deeply supervised structure makes the training much easier and also produces better classification results on ImageNet and the recently released, larger MIT Places dataset