# MoPro: Webly Supervised Learning with Momentum Prototypes

arXiv: https://arxiv.org/abs/2009.07995 · HTML: https://mlsanity.com/paper/2009.07995/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 30 of 66)

- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Non-Parametric Outlier Synthesis](https://mlsanity.com/paper/2303.02966/) (2023)
- [How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?](https://mlsanity.com/paper/2203.04450/) (2022)
- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [NGC: A Unified Framework for Learning with Open-World Noisy Data](https://mlsanity.com/paper/2108.11035/) (2021)
- [Learning with Neighbor Consistency for Noisy Labels](https://mlsanity.com/paper/2202.02200/) (2022)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [DASO: Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2106.05682/) (2021)
- [Learning with Mixture of Prototypes for Out-of-Distribution Detection](https://mlsanity.com/paper/2402.02653/) (2024)
- [Large-Scale Pre-training for Person Re-identification with Noisy Labels](https://mlsanity.com/paper/2203.16533/) (2022)
- [Benchmarking Omni-Vision Representation through the Lens of Visual Realms](https://mlsanity.com/paper/2207.07106/) (2022)
- [Robust Long-Tailed Learning under Label Noise](https://mlsanity.com/paper/2108.11569/) (2021)
- [OpenCon: Open-world Contrastive Learning](https://mlsanity.com/paper/2208.02764/) (2022)
- [Twin Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2303.06930/) (2023)
- [Continual Learning on Noisy Data Streams via Self-Purified Replay](https://mlsanity.com/paper/2110.07735/) (2021)
- [Towards Federated Learning against Noisy Labels via Local Self-Regularization](https://mlsanity.com/paper/2208.12807/) (2022)
- [PiCO+: Contrastive Label Disambiguation for Robust Partial Label Learning](https://mlsanity.com/paper/2201.08984/) (2022)
- [On Learning Contrastive Representations for Learning with Noisy Labels](https://mlsanity.com/paper/2203.01785/) (2022)
- [Multi-label Iterated Learning for Image Classification with Label Ambiguity](https://mlsanity.com/paper/2111.12172/) (2021)
- [Hyperspherical Consistency Regularization](https://mlsanity.com/paper/2206.00845/) (2022)
- [Fine-Grained Classification with Noisy Labels](https://mlsanity.com/paper/2303.02404/) (2023)
- [Masked Unsupervised Self-training for Label-free Image Classification](https://mlsanity.com/paper/2206.02967/) (2022)
- [Improving Zero-Shot Models with Label Distribution Priors](https://mlsanity.com/paper/2212.00784/) (2022)
- [Contrastive Attraction and Contrastive Repulsion for Representation Learning](https://mlsanity.com/paper/2105.03746/) (2021)
- [Binary Classification with Confidence Difference](https://mlsanity.com/paper/2310.05632/) (2023)
- [HYPO: Hyperspherical Out-of-Distribution Generalization](https://mlsanity.com/paper/2402.07785/) (2024)
- [MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task](https://mlsanity.com/paper/2402.18603/) (2024)
- [Exploring Visual Engagement Signals for Representation Learning](https://mlsanity.com/paper/2104.07767/) (2021)

## Abstract (arXiv)

We propose a webly-supervised representation learning method that does not suffer from the annotation unscalability of supervised learning, nor the computation unscalability of self-supervised learning. Most existing works on webly-supervised representation learning adopt a vanilla supervised learning method without accounting for the prevalent noise in the training data, whereas most prior methods in learning with label noise are less effective for real-world large-scale noisy data. We propose momentum prototypes (MoPro), a simple contrastive learning method that achieves online label noise correction, out-of-distribution sample removal, and representation learning. MoPro achieves state-of-the-art performance on WebVision, a weakly-labeled noisy dataset. MoPro also shows superior performance when the pretrained model is transferred to down-stream image classification and detection tasks. It outperforms the ImageNet supervised pretrained model by +10.5 on 1-shot classification on VOC, and outperforms the best self-supervised pretrained model by +17.3 when finetuned on 1% of ImageNet labeled samples. Furthermore, MoPro is more robust to distribution shifts. Code and pretrained models are available at https://github.com/salesforce/MoPro.