# Humble Teachers Teach Better Students for Semi-Supervised Object Detection

arXiv: https://arxiv.org/abs/2106.10456 · HTML: https://mlsanity.com/paper/2106.10456/

## Key facts

- Cited by 116 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 41 of its 60 references that are in the map.
- First posted on arXiv 19 Jun 2021.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring](https://mlsanity.com/paper/1911.09785/) (2019)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)

## Built on by (most cited first, 30 of 116)

- [Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.09910/) (2022)
- [USB: A Unified Semi-supervised Learning Benchmark for Classification](https://mlsanity.com/paper/2208.07204/) (2022)
- [Unbiased Teacher v2: Semi-supervised Object Detection for Anchor-free and Anchor-based Detectors](https://mlsanity.com/paper/2206.09500/) (2022)
- [Class-Aware Contrastive Semi-Supervised Learning](https://mlsanity.com/paper/2203.02261/) (2022)
- [UniMatch V2: Pushing the Limit of Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2410.10777/) (2024)
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](https://mlsanity.com/paper/2208.11296/) (2022)
- [PseCo: Pseudo Labeling and Consistency Training for Semi-Supervised Object Detection](https://mlsanity.com/paper/2203.16317/) (2022)
- [Dense Teacher: Dense Pseudo-Labels for Semi-supervised Object Detection](https://mlsanity.com/paper/2207.02541/) (2022)
- [Consistent-Teacher: Towards Reducing Inconsistent Pseudo-targets in Semi-supervised Object Detection](https://mlsanity.com/paper/2209.01589/) (2022)
- [Active Teacher for Semi-Supervised Object Detection](https://mlsanity.com/paper/2303.08348/) (2023)
- [Dense Learning based Semi-Supervised Object Detection](https://mlsanity.com/paper/2204.07300/) (2022)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Omni-DETR: Omni-Supervised Object Detection with Transformers](https://mlsanity.com/paper/2203.16089/) (2022)
- [TeST: Test-time Self-Training under Distribution Shift](https://mlsanity.com/paper/2209.11459/) (2022)
- [Semi-Supervised Object Detection with Adaptive Class-Rebalancing Self-Training](https://mlsanity.com/paper/2107.05031/) (2021)
- [Audio-Visual Segmentation via Unlabeled Frame Exploitation](https://mlsanity.com/paper/2403.11074/) (2024)
- [Not All Labels Are Equal: Rationalizing The Labeling Costs for Training Object Detection](https://mlsanity.com/paper/2106.11921/) (2021)
- [Hierarchical Supervision and Shuffle Data Augmentation for 3D Semi-Supervised Object Detection](https://mlsanity.com/paper/2304.01464/) (2023)
- [Mix-Teaching: A Simple, Unified and Effective Semi-Supervised Learning Framework for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.04448/) (2022)
- [Noisy Boundaries: Lemon or Lemonade for Semi-supervised Instance Segmentation?](https://mlsanity.com/paper/2203.13427/) (2022)
- [DetMatch: Two Teachers are Better Than One for Joint 2D and 3D Semi-Supervised Object Detection](https://mlsanity.com/paper/2203.09510/) (2022)
- [SOOD: Towards Semi-Supervised Oriented Object Detection](https://mlsanity.com/paper/2304.04515/) (2023)
- [Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification](https://mlsanity.com/paper/2111.00928/) (2021)
- [DDS3D: Dense Pseudo-Labels with Dynamic Threshold for Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/2303.05079/) (2023)
- [Doubly Robust Self-Training](https://mlsanity.com/paper/2306.00265/) (2023)
- [End-to-End Semi-Supervised Learning for Video Action Detection](https://mlsanity.com/paper/2203.04251/) (2022)
- [Open-Set Semi-Supervised Object Detection](https://mlsanity.com/paper/2208.13722/) (2022)
- [Efficient Teacher: Semi-Supervised Object Detection for YOLOv5](https://mlsanity.com/paper/2302.07577/) (2023)
- [Pseudo-label Alignment for Semi-supervised Instance Segmentation](https://mlsanity.com/paper/2308.05359/) (2023)
- [Label-Efficient Online Continual Object Detection in Streaming Video](https://mlsanity.com/paper/2206.00309/) (2022)

## Abstract (arXiv)

We propose a semi-supervised approach for contemporary object detectors following the teacher-student dual model framework. Our method is featured with 1) the exponential moving averaging strategy to update the teacher from the student online, 2) using plenty of region proposals and soft pseudo-labels as the student's training targets, and 3) a light-weighted detection-specific data ensemble for the teacher to generate more reliable pseudo-labels. Compared to the recent state-of-the-art -- STAC, which uses hard labels on sparsely selected hard pseudo samples, the teacher in our model exposes richer information to the student with soft-labels on many proposals. Our model achieves COCO-style AP of 53.04% on VOC07 val set, 8.4% better than STAC, when using VOC12 as unlabeled data. On MS-COCO, it outperforms prior work when only a small percentage of data is taken as labeled. It also reaches 53.8% AP on MS-COCO test-dev with 3.1% gain over the fully supervised ResNet-152 Cascaded R-CNN, by tapping into unlabeled data of a similar size to the labeled data.