# Improving Fast Segmentation With Teacher-student Learning

arXiv: https://arxiv.org/abs/1810.08476 · HTML: https://mlsanity.com/paper/1810.08476/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2018.

## Built on by (most cited first, 30 of 34)

- [Cross-Image Relational Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2204.06986/) (2022)
- [SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2108.06227/) (2021)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [Efficient Medical Image Segmentation Based on Knowledge Distillation](https://mlsanity.com/paper/2108.09987/) (2021)
- [EdgeSAM: Prompt-In-the-Loop Distillation for SAM](https://mlsanity.com/paper/2312.06660/) (2023)
- [Structural and Statistical Texture Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2305.03944/) (2023)
- [LLAVADI: What Matters For Multimodal Large Language Models Distillation](https://mlsanity.com/paper/2407.19409/) (2024)
- [Double Similarity Distillation for Semantic Image Segmentation](https://mlsanity.com/paper/2107.08591/) (2021)
- [TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentation](https://mlsanity.com/paper/2202.13393/) (2022)
- [Distilling Inter-Class Distance for Semantic Segmentation](https://mlsanity.com/paper/2205.03650/) (2022)
- [Pixel-by-Pixel Cross-Domain Alignment for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2110.11650/) (2021)
- [Exploring the Application of Large-scale Pre-trained Models on Adverse Weather Removal](https://mlsanity.com/paper/2306.09008/) (2023)
- [SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment](https://mlsanity.com/paper/2509.03934/) (2025)
- [MS-KD: Multi-Organ Segmentation with Multiple Binary-Labeled Datasets](https://mlsanity.com/paper/2108.02559/) (2021)
- [Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfer](https://mlsanity.com/paper/2104.14696/) (2021)
- [Facial Landmark Points Detection Using Knowledge Distillation-Based Neural Networks](https://mlsanity.com/paper/2111.07047/) (2021)
- [Multi-Granularity Distillation Scheme Towards Lightweight Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.10169/) (2022)
- [Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data](https://mlsanity.com/paper/2103.13733/) (2021)
- [WeClick: Weakly-Supervised Video Semantic Segmentation with Click Annotations](https://mlsanity.com/paper/2107.03088/) (2021)
- [Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation](https://mlsanity.com/paper/2110.09674/) (2021)
- [U-HRNet: Delving into Improving Semantic Representation of High Resolution Network for Dense Prediction](https://mlsanity.com/paper/2210.07140/) (2022)
- [Attention-guided Feature Distillation for Semantic Segmentation](https://mlsanity.com/paper/2403.05451/) (2024)
- [Vision-Based Detection of Uncooperative Targets and Components on Small Satellites](https://mlsanity.com/paper/2408.12084/) (2024)
- [Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network](https://mlsanity.com/paper/2103.00793/) (2021)
- [Similarity Transfer for Knowledge Distillation](https://mlsanity.com/paper/2103.10047/) (2021)
- [Normalized Feature Distillation for Semantic Segmentation](https://mlsanity.com/paper/2207.05256/) (2022)
- [AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentation](https://mlsanity.com/paper/2308.04243/) (2023)
- [Rich Feature Distillation with Feature Affinity Module for Efficient Image Dehazing](https://mlsanity.com/paper/2207.11250/) (2022)
- [Towards Comparable Knowledge Distillation in Semantic Image Segmentation](https://mlsanity.com/paper/2309.03659/) (2023)
- [Effectiveness of Function Matching in Driving Scene Recognition](https://mlsanity.com/paper/2208.09694/) (2022)

## Abstract (arXiv)

Recently, segmentation neural networks have been significantly improved by demonstrating very promising accuracies on public benchmarks. However, these models are very heavy and generally suffer from low inference speed, which limits their application scenarios in practice. Meanwhile, existing fast segmentation models usually fail to obtain satisfactory segmentation accuracies on public benchmarks. In this paper, we propose a teacher-student learning framework that transfers the knowledge gained by a heavy and better performed segmentation network (i.e. teacher) to guide the learning of fast segmentation networks (i.e. student). Specifically, both zero-order and first-order knowledge depicted in the fine annotated images and unlabeled auxiliary data are transferred to regularize our student learning. The proposed method can improve existing fast segmentation models without incurring extra computational overhead, so it can still process images with the same fast speed. Extensive experiments on the Pascal Context, Cityscape and VOC 2012 datasets demonstrate that the proposed teacher-student learning framework is able to significantly boost the performance of student network.