# Billion-scale semi-supervised learning for image classification

arXiv: https://arxiv.org/abs/1905.00546 · HTML: https://mlsanity.com/paper/1905.00546/

## Key facts

- Cited by 232 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2019.

## Built on by (most cited first, 30 of 232)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack](https://mlsanity.com/paper/2309.15807/) (2023)
- [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2106.05095/) (2021)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [Offboard 3D Object Detection from Point Cloud Sequences](https://mlsanity.com/paper/2103.05073/) (2021)
- [Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline](https://mlsanity.com/paper/2106.05304/) (2021)
- [Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization](https://mlsanity.com/paper/2110.10832/) (2021)
- [Unsupervised Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2204.03649/) (2022)
- [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [On the Adversarial Robustness of Vision Transformers](https://mlsanity.com/paper/2103.15670/) (2021)
- [Compressing Visual-linguistic Model via Knowledge Distillation](https://mlsanity.com/paper/2104.02096/) (2021)
- [Multi-Task Self-Training for Learning General Representations](https://mlsanity.com/paper/2108.11353/) (2021)
- [Self-Supervised Pretraining Improves Self-Supervised Pretraining](https://mlsanity.com/paper/2103.12718/) (2021)
- [Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels](https://mlsanity.com/paper/2103.13646/) (2021)
- [Compute and Energy Consumption Trends in Deep Learning Inference](https://mlsanity.com/paper/2109.05472/) (2021)
- [Continual Learning with Foundation Models: An Empirical Study of Latent Replay](https://mlsanity.com/paper/2205.00329/) (2022)
- [Data Feedback Loops: Model-driven Amplification of Dataset Biases](https://mlsanity.com/paper/2209.03942/) (2022)

## Abstract (arXiv)

This paper presents a study of semi-supervised learning with large convolutional networks. We propose a pipeline, based on a teacher/student paradigm, that leverages a large collection of unlabelled images (up to 1 billion). Our main goal is to improve the performance for a given target architecture, like ResNet-50 or ResNext. We provide an extensive analysis of the success factors of our approach, which leads us to formulate some recommendations to produce high-accuracy models for image classification with semi-supervised learning. As a result, our approach brings important gains to standard architectures for image, video and fine-grained classification. For instance, by leveraging one billion unlabelled images, our learned vanilla ResNet-50 achieves 81.2% top-1 accuracy on the ImageNet benchmark.