# Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches

arXiv: https://arxiv.org/abs/2207.08220 · HTML: https://mlsanity.com/paper/2207.08220/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 33 references that are in the map.
- First posted on arXiv 17 Jul 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning](https://mlsanity.com/paper/2012.11552/) (2020)
- [Mean Shift for Self-Supervised Learning](https://mlsanity.com/paper/2105.07269/) (2021)
- [Revisiting the Transferability of Supervised Pretraining: an MLP Perspective](https://mlsanity.com/paper/2112.00496/) (2021)
- [Jigsaw Clustering for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2104.00323/) (2021)

## Built on by (most cited first, 16 of 16)

- [Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems](https://mlsanity.com/paper/2303.01669/) (2023)
- [FreCT: Frequency-augmented Convolutional Transformer for Robust Time Series Anomaly Detection](https://mlsanity.com/paper/2505.00941/) (2025)
- [Pseudo-Label Calibration Semi-supervised Multi-Modal Entity Alignment](https://mlsanity.com/paper/2403.01203/) (2024)
- [Non-Neighbors Also Matter to Kriging: A New Contrastive-Prototypical Learning](https://mlsanity.com/paper/2401.12681/) (2024)
- [Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning](https://mlsanity.com/paper/2409.10362/) (2024)
- [Self-Supervised Learning for Image Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2505.13584/) (2025)
- [CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation](https://mlsanity.com/paper/2504.21478/) (2025)
- [On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning](https://mlsanity.com/paper/2404.19289/) (2024)
- [Accelerating Augmentation Invariance Pretraining](https://mlsanity.com/paper/2410.22364/) (2024)
- [Deformable Attentive Visual Enhancement for Referring Segmentation Using Vision-Language Model](https://mlsanity.com/paper/2505.19242/) (2025)
- [Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2306.01929/) (2023)
- [Multi-network Contrastive Learning Based on Global and Local Representations](https://mlsanity.com/paper/2306.15930/) (2023)
- [Content-decoupled Contrastive Learning-based Implicit Degradation Modeling for Blind Image Super-Resolution](https://mlsanity.com/paper/2408.05440/) (2024)
- [Taming the Randomness: Towards Label-Preserving Cropping in Contrastive Learning](https://mlsanity.com/paper/2504.19824/) (2025)
- [Understanding InfoNCE: Transition Probability Matrix Induced Feature Clustering](https://mlsanity.com/paper/2511.12180/) (2025)
- [Depth-Guided Contrastive Learning for 2D Representations with 3D Spatial Awareness](https://mlsanity.com/paper/2609.28159/) (2026)

## Abstract (arXiv)

Contrastive-based self-supervised learning methods achieved great success in recent years. However, self-supervision requires extremely long training epochs (e.g., 800 epochs for MoCo v3) to achieve promising results, which is unacceptable for the general academic community and hinders the development of this topic. This work revisits the momentum-based contrastive learning frameworks and identifies the inefficiency in which two augmented views generate only one positive pair. We propose Fast-MoCo - a novel framework that utilizes combinatorial patches to construct multiple positive pairs from two augmented views, which provides abundant supervision signals that bring significant acceleration with neglectable extra computational cost. Fast-MoCo trained with 100 epochs achieves 73.5% linear evaluation accuracy, similar to MoCo v3 (ResNet-50 backbone) trained with 800 epochs. Extra training (200 epochs) further improves the result to 75.1%, which is on par with state-of-the-art methods. Experiments on several downstream tasks also confirm the effectiveness of Fast-MoCo.