mlsanityData to 14 Oct 2026

Temporal Ensembling for Semi-Supervised Learning

arXiv 1610.02242 ↗First posted 7 Oct 2016cs.NE cs.LGcited by 1,057

Not on the map. Why?

Cited by 47 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce self-ensembling, where we form a consensus prediction of the unknown labels using the outputs of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the (non-augmented) classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from 18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16% by enabling the standard augmentations. We additionally obtain a clear improvement in CIFAR-100 classification accuracy by using random images from the Tiny Images dataset as unlabeled extra inputs during training. Finally, we demonstrate good tolerance to incorrect labels.

Key facts. Cited by 1,057 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Oct 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 267; 2022: 241; 2023: 217; 2024: 173; 2025: 99; 2026: 60.

1,057 papers.

Sort:

By area:

  1. 2022Diffusion Models in Vision: A Surveycited by 1,015
  2. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  4. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 463
  5. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  6. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  7. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 316
  8. 2021Unbiased Teacher for Semi-Supervised Object Detectioncited by 306
  9. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  10. 2021A Fourier-based Framework for Domain Generalizationcited by 290
  11. 2021Ensemble deep learning: A reviewcited by 284
  12. 2021A Survey on Deep Semi-supervised Learningcited by 251
38 more
  1. 2022FreeMatch: Self-adaptive Thresholding for Semi-supervised Learningcited by 232
  2. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  3. 2022Generalized Category Discoverycited by 211
  4. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  5. 2021ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentationcited by 184
  6. 2022Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentationcited by 178
  7. 2021CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learningcited by 175
  8. 2021Open-World Semi-Supervised Learningcited by 165
  9. 2021Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Datacited by 152
  10. 2021Self-supervised Augmentation Consistency for Adapting Semantic Segmentationcited by 149
  11. 2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 148
  12. 2021Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizationcited by 141
  13. 2023SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learningcited by 133
  14. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  15. 2022SimMatch: Semi-supervised Learning with Similarity Matchingcited by 118
  16. 2021Humble Teachers Teach Better Students for Semi-Supervised Object Detectioncited by 116
  17. 2021AutoNovel: Automatically Discovering and Learning Novel Visual Categoriescited by 116
  18. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  19. 2021Instant-Teaching: An End-to-End Semi-Supervised Object Detection Frameworkcited by 113
  20. 2021Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentationcited by 113
  21. 2021Semi-supervised Semantic Segmentation with Directional Context-aware Consistencycited by 112
  22. 2021Mutual Consistency Learning for Semi-supervised Medical Image Segmentationcited by 108
  23. 2021Regularizing Generative Adversarial Networks under Limited Datacited by 107
  24. 2021Semi-supervised Left Atrium Segmentation with Mutual Consistency Trainingcited by 104
  25. 2021SPICE: Semantic Pseudo-labeling for Image Clusteringcited by 104
  26. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  27. 2021AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptationcited by 104
  28. 2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
  29. 2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
  30. 2021Semi-Supervised Semantic Segmentation via Adaptive Equalization Learningcited by 101
  31. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  32. 2021Unsupervised Domain Adaptation of Object Detectors: A Surveycited by 93
  33. 2022Exploring Smoothness and Class-Separation for Semi-supervised Medical Image Segmentationcited by 93
  34. 2022LaserMix for Semi-Supervised LiDAR Semantic Segmentationcited by 93
  35. 2023Deep Model Fusion: A Surveycited by 91
  36. 2021Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognitioncited by 89
  37. 2021One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentationcited by 89
  38. 2021Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentationcited by 87

50 most cited of 1,057 All 1,057 (JSON)