# Deeply-Supervised Nets

arXiv: https://arxiv.org/abs/1409.5185 · HTML: https://mlsanity.com/paper/1409.5185/

## Key facts

- Cited by 367 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2014.

## Built on by (most cited first, 30 of 367)

- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining](https://mlsanity.com/paper/2402.03302/) (2024)
- [Highly Accurate Dichotomous Image Segmentation](https://mlsanity.com/paper/2203.03041/) (2022)
- [Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity](https://mlsanity.com/paper/2111.14330/) (2021)
- [MOOD: Multi-level Out-of-distribution Detection](https://mlsanity.com/paper/2104.14726/) (2021)
- [PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery](https://mlsanity.com/paper/2212.05590/) (2022)
- [Robustness to Unbounded Smoothness of Generalized SignSGD](https://mlsanity.com/paper/2208.11195/) (2022)
- [Line Segment Detection Using Transformers without Edges](https://mlsanity.com/paper/2101.01909/) (2021)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [MPCFormer: fast, performant and private Transformer inference with MPC](https://mlsanity.com/paper/2211.01452/) (2022)
- [EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism](https://mlsanity.com/paper/2312.04916/) (2023)
- [MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption](https://mlsanity.com/paper/2103.16201/) (2021)
- [End-to-End Test-Time Training for Long Context](https://mlsanity.com/paper/2512.23675/) (2025)
- [FrameExit: Conditional Early Exiting for Efficient Video Recognition](https://mlsanity.com/paper/2104.13400/) (2021)
- [M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2303.05302/) (2023)
- [MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation](https://mlsanity.com/paper/2105.07451/) (2021)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers](https://mlsanity.com/paper/2107.12636/) (2021)
- [Advances in Deep Concealed Scene Understanding](https://mlsanity.com/paper/2304.11234/) (2023)
- [Understanding AdamW through Proximal Methods and Scale-Freeness](https://mlsanity.com/paper/2202.00089/) (2022)
- [Boundary-Aware Segmentation Network for Mobile and Web Applications](https://mlsanity.com/paper/2101.04704/) (2021)
- [Local-Global Context Aware Transformer for Language-Guided Video Segmentation](https://mlsanity.com/paper/2203.09773/) (2022)
- [DRCT: Saving Image Super-resolution away from Information Bottleneck](https://mlsanity.com/paper/2404.00722/) (2024)
- [Every Annotation Counts: Multi-label Deep Supervision for Medical Image Segmentation](https://mlsanity.com/paper/2104.13243/) (2021)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [Video-based Person Re-identification with Spatial and Temporal Memory Networks](https://mlsanity.com/paper/2108.09039/) (2021)
- [MTTrans: Cross-Domain Object Detection with Mean-Teacher Transformer](https://mlsanity.com/paper/2205.01643/) (2022)

## Abstract (arXiv)

Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent. We make an attempt to boost the classification performance by studying a new formulation in deep networks. Three aspects in convolutional neural networks (CNN) style architectures are being looked at: (1) transparency of the intermediate layers to the overall classification; (2) discriminativeness and robustness of learned features, especially in the early layers; (3) effectiveness in training due to the presence of the exploding and vanishing gradients. We introduce "companion objective" to the individual hidden layers, in addition to the overall objective at the output layer (a different strategy to layer-wise pre-training). We extend techniques from stochastic gradient methods to analyze our algorithm. The advantage of our method is evident and our experimental result on benchmark datasets shows significant performance gain over existing methods (e.g. all state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and SVHN).