mlsanityData to 14 Oct 2026

Self-training with Noisy Student improves ImageNet classification

arXiv 1911.04252 ↗First posted 11 Nov 2019cs.LG cs.CV stat.MLcited by 1,311

Not on the map. Why?

Cited by 98 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Noisy Student Training, a semi-supervised learning approach that works well even when labeled data is abundant. Noisy Student Training achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2. Noisy Student Training extends the idea of self-training and distillation with the use of equal-or-larger student models and noise added to the student during learning. On ImageNet, we first train an EfficientNet model on labeled images and use it as a teacher to generate pseudo labels for 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. We iterate this process by putting back the student as the teacher. During the learning of the student, we inject noise such as dropout, stochastic depth, and data augmentation via RandAugment to the student so that the student generalizes better than the teacher. Models are available at https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet. Code is available at https://github.com/google-research/noisystudent.

Key facts. Cited by 1,311 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 351; 2022: 301; 2023: 275; 2024: 184; 2025: 106; 2026: 93.

1,311 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  7. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  8. 2021Scaling Vision Transformerscited by 999
  9. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  11. 2021Robust fine-tuning of zero-shot modelscited by 797
  12. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
38 more
  1. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  2. 2022Large Language Models Can Self-Improvecited by 664
  3. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  4. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  5. 2021ImageNet-21K Pretraining for the Massescited by 613
  6. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  7. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  8. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  9. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  11. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 463
  12. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  13. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  14. 2026Reinforcement Learning via Self-Distillationcited by 332
  15. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  16. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 316
  17. 2021Unbiased Teacher for Semi-Supervised Object Detectioncited by 306
  18. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  19. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  20. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  21. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  22. 2021Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentationcited by 268
  23. 2023Towards Unbounded Machine Unlearningcited by 268
  24. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  25. 2021A Survey on Deep Semi-supervised Learningcited by 251
  26. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  27. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  28. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  29. 2022FreeMatch: Self-adaptive Thresholding for Semi-supervised Learningcited by 232
  30. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  31. 2021WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognitioncited by 223
  32. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 221
  33. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  34. 2022Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labelscited by 218
  35. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  36. 2021Vision Transformers are Robust Learnerscited by 207
  37. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  38. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194

50 most cited of 1,311 All 1,311 (JSON)