# Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentation

arXiv: https://arxiv.org/abs/2407.13254 · HTML: https://mlsanity.com/paper/2407.13254/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 43 of its 57 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 43)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Dataset Distillation with Infinitely Wide Convolutional Networks](https://mlsanity.com/paper/2107.13034/) (2021)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Channel-wise Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/2011.13256/) (2020)
- [Learning from Noisy Labels with Distillation](https://mlsanity.com/paper/1703.02391/) (2017)
- [2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds](https://mlsanity.com/paper/2207.04397/) (2022)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Rethinking BiSeNet For Real-time Semantic Segmentation](https://mlsanity.com/paper/2104.13188/) (2021)
- [Cross-Image Relational Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2204.06986/) (2022)
- [Knowledge Adaptation for Efficient Semantic Segmentation](https://mlsanity.com/paper/1903.04688/) (2019)
- [DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1904.02216/) (2019)
- [VITA: Video Instance Segmentation via Object Token Association](https://mlsanity.com/paper/2206.04403/) (2022)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)

## Built on by (most cited first, 5 of 5)

- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [DUDA: Distilled Unsupervised Domain Adaptation for Lightweight Semantic Segmentation](https://mlsanity.com/paper/2504.09814/) (2025)
- [AMap: Distilling Future Priors for Ahead-Aware Online HD Map Construction](https://mlsanity.com/paper/2512.19150/) (2025)
- [MAFS: Masked Autoencoder for Infrared-Visible Image Fusion and Semantic Segmentation](https://mlsanity.com/paper/2509.11817/) (2025)
- [CasPoinTr: Point Cloud Completion with Cascaded Networks and Knowledge Distillation](https://mlsanity.com/paper/2509.23375/) (2025)

## Abstract (arXiv)

In this paper, we introduce a novel knowledge distillation approach for the semantic segmentation task. Unlike previous methods that rely on power-trained teachers or other modalities to provide additional knowledge, our approach does not require complex teacher models or information from extra sensors. Specifically, for the teacher model training, we propose to noise the label and then incorporate it into input to effectively boost the lightweight teacher performance. To ensure the robustness of the teacher model against the introduced noise, we propose a dual-path consistency training strategy featuring a distance loss between the outputs of two paths. For the student model training, we keep it consistent with the standard distillation for simplicity. Our approach not only boosts the efficacy of knowledge distillation but also increases the flexibility in selecting teacher and student models. To demonstrate the advantages of our Label Assisted Distillation (LAD) method, we conduct extensive experiments on five challenging datasets including Cityscapes, ADE20K, PASCAL-VOC, COCO-Stuff 10K, and COCO-Stuff 164K, five popular models: FCN, PSPNet, DeepLabV3, STDC, and OCRNet, and results show the effectiveness and generalization of our approach. We posit that incorporating labels into the input, as demonstrated in our work, will provide valuable insights into related fields. Code is available at https://github.com/skyshoumeng/Label_Assisted_Distillation.