# Switching Temporary Teachers for Semi-Supervised Semantic Segmentation

arXiv: https://arxiv.org/abs/2310.18640 · HTML: https://mlsanity.com/paper/2310.18640/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 42 of its 56 references that are in the map.
- First posted on arXiv 28 Oct 2023.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision](https://mlsanity.com/paper/2106.01226/) (2021)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Unbiased Teacher for Semi-Supervised Object Detection](https://mlsanity.com/paper/2102.09480/) (2021)
- [DACS: Domain Adaptation via Cross-domain Mixed Sampling](https://mlsanity.com/paper/2007.08702/) (2020)
- [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](https://mlsanity.com/paper/2203.03884/) (2022)
- [ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning](https://mlsanity.com/paper/2007.07936/) (2020)
- [ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2106.05095/) (2021)
- [PseudoSeg: Designing Pseudo Labels for Semantic Segmentation](https://mlsanity.com/paper/2010.09713/) (2020)
- [Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.09910/) (2022)
- [Instance Adaptive Self-Training for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2008.12197/) (2020)

## Built on by (most cited first, 23 of 23)

- [Alternate Diverse Teaching for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2311.17325/) (2023)
- [SemiCD-VL: Visual-Language Model Guidance Makes Better Semi-supervised Change Detector](https://mlsanity.com/paper/2405.04788/) (2024)
- [D3T: Distinctive Dual-Domain Teacher Zigzagging Across RGB-Thermal Gap for Domain-Adaptive Object Detection](https://mlsanity.com/paper/2403.09359/) (2024)
- [SDCL: Students Discrepancy-Informed Correction Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2409.16728/) (2024)
- [PrevMatch: Revisiting and Maximizing Temporal Knowledge in Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2405.20610/) (2024)
- [PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2409.05122/) (2024)
- [Feedback-Driven Pseudo-Label Reliability Assessment: Redefining Thresholding for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2505.07691/) (2025)
- [HDC: Hierarchical Distillation for Multi-level Noisy Consistency in Semi-Supervised Fetal Ultrasound Segmentation](https://mlsanity.com/paper/2504.09876/) (2025)
- [Rethinking Semi-supervised Segmentation Beyond Accuracy: Reliability and Robustness](https://mlsanity.com/paper/2506.05917/) (2025)
- [A Point-Neighborhood Learning Framework for Nasal Endoscope Image Segmentation](https://mlsanity.com/paper/2405.20044/) (2024)
- [IGL-DT: Iterative Global-Local Feature Learning with Dual-Teacher Semantic Segmentation Framework under Limited Annotation Scheme](https://mlsanity.com/paper/2504.09797/) (2025)
- [Adaptive Knowledge Transferring with Switching Dual-Student Framework for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2510.24366/) (2025)
- [CTFS : Collaborative Teacher Framework for Forward-Looking Sonar Image Semantic Segmentation with Extremely Limited Labels](https://mlsanity.com/paper/2603.21071/) (2026)
- [Less is More: Pseudo-Label Filtering for Continual Test-Time Adaptation](https://mlsanity.com/paper/2406.02609/) (2024)
- [Depth-Guided Semi-Supervised Instance Segmentation](https://mlsanity.com/paper/2406.17413/) (2024)
- [Exploring Token-Level Augmentation in Vision Transformer for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2503.02459/) (2025)
- [RAIL: Region-Aware Instructive Learning for Semi-Supervised Tooth Segmentation in CBCT](https://mlsanity.com/paper/2505.03538/) (2025)
- [When Noise Meets Long-Tail: Feature-Threshold Dual Calibration for Robust Pseudo-Labeling](https://mlsanity.com/paper/2609.33668/) (2026)
- [Dual-Teacher Ensemble Models with Double-Copy-Paste for 3D Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2410.11509/) (2024)
- [BoundMatch: Boundary detection applied to semi-supervised segmentation](https://mlsanity.com/paper/2503.23519/) (2025)
- [SemiISP/SemiIE: Semi-Supervised Image Signal Processor and Image Enhancement Leveraging One-to-Many Mapping sRGB-to-RAW](https://mlsanity.com/paper/2504.02345/) (2025)
- [Dual Teacher-Student Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2505.11018/) (2025)
- [Leveraging Out-of-Distribution Unlabeled Images: Semi-Supervised Semantic Segmentation with an Open-Vocabulary Model](https://mlsanity.com/paper/2507.03302/) (2025)

## Abstract (arXiv)

The teacher-student framework, prevalent in semi-supervised semantic segmentation, mainly employs the exponential moving average (EMA) to update a single teacher's weights based on the student's. However, EMA updates raise a problem in that the weights of the teacher and student are getting coupled, causing a potential performance bottleneck. Furthermore, this problem may become more severe when training with more complicated labels such as segmentation masks but with few annotated data. This paper introduces Dual Teacher, a simple yet effective approach that employs dual temporary teachers aiming to alleviate the coupling problem for the student. The temporary teachers work in shifts and are progressively improved, so consistently prevent the teacher and student from becoming excessively close. Specifically, the temporary teachers periodically take turns generating pseudo-labels to train a student model and maintain the distinct characteristics of the student model for each epoch. Consequently, Dual Teacher achieves competitive performance on the PASCAL VOC, Cityscapes, and ADE20K benchmarks with remarkably shorter training times than state-of-the-art methods. Moreover, we demonstrate that our approach is model-agnostic and compatible with both CNN- and Transformer-based models. Code is available at urlhttps://github.com/naver-ai/dual-teacher.