# Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

arXiv: https://arxiv.org/abs/2207.06167 · HTML: https://mlsanity.com/paper/2207.06167/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 76 references that are in the map.
- First posted on arXiv 13 Jul 2022.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)

## Built on by (most cited first, 22 of 22)

- [Diffusion Models Beat GANs on Image Classification](https://mlsanity.com/paper/2307.08702/) (2023)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [Do text-free diffusion models learn discriminative visual representations?](https://mlsanity.com/paper/2311.17921/) (2023)
- [LDReg: Local Dimensionality Regularized Self-Supervised Learning](https://mlsanity.com/paper/2401.10474/) (2024)
- [Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization](https://mlsanity.com/paper/2411.00392/) (2024)
- [Revisiting Foreground and Background Separation in Weakly-supervised Temporal Action Localization: A Clustering-based Approach](https://mlsanity.com/paper/2312.14138/) (2023)
- [Feudal Networks for Visual Navigation](https://mlsanity.com/paper/2402.12498/) (2024)
- [Calibre: Towards Fair and Accurate Personalized Federated Learning with Self-Supervised Learning](https://mlsanity.com/paper/2412.20020/) (2024)
- [Cluster Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2507.12359/) (2025)
- [Ladder Siamese Network: a Method and Insights for Multi-level Self-Supervised Learning](https://mlsanity.com/paper/2211.13844/) (2022)
- [Self-Supervised Learning Based Handwriting Verification](https://mlsanity.com/paper/2405.18320/) (2024)
- [Efficient Unsupervised Visual Representation Learning with Explicit Cluster Balancing](https://mlsanity.com/paper/2407.11168/) (2024)
- [POA: Pre-training Once for Models of All Sizes](https://mlsanity.com/paper/2408.01031/) (2024)
- [Enhancing Contrastive Learning Inspired by the Philosophy of "The Blind Men and the Elephant"](https://mlsanity.com/paper/2412.16522/) (2024)
- [Implicit Neural Representation Facilitates Unified Universal Vision Encoding](https://mlsanity.com/paper/2601.14256/) (2026)
- [ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives](https://mlsanity.com/paper/2609.01041/) (2026)
- [Removing supervision in semantic segmentation with local-global matching and area balancing](https://mlsanity.com/paper/2303.17410/) (2023)
- [Towards Efficient and Effective Deep Clustering with Dynamic Grouping and Prototype Aggregation](https://mlsanity.com/paper/2401.13581/) (2024)
- [The Bad Batches: Enhancing Self-Supervised Learning in Image Classification Through Representative Batch Curation](https://mlsanity.com/paper/2403.19579/) (2024)
- [Taming the Randomness: Towards Label-Preserving Cropping in Contrastive Learning](https://mlsanity.com/paper/2504.19824/) (2025)
- [FeudalNav: A Simple Framework for Visual Navigation](https://mlsanity.com/paper/2602.06974/) (2026)
- [KCLNet: Electrically Equivalence-Oriented Graph Representation Learning for Analog Circuits](https://mlsanity.com/paper/2603.24101/) (2026)

## Abstract (arXiv)

In this paper, we propose a genuine group-level contrastive visual representation learning method whose linear evaluation performance on ImageNet surpasses the vanilla supervised learning. Two mainstream unsupervised learning schemes are the instance-level contrastive framework and clustering-based schemes. The former adopts the extremely fine-grained instance-level discrimination whose supervisory signal is not efficient due to the false negatives. Though the latter solves this, they commonly come with some restrictions affecting the performance. To integrate their advantages, we design the SMoG method. SMoG follows the framework of contrastive learning but replaces the contrastive unit from instance to group, mimicking clustering-based methods. To achieve this, we propose the momentum grouping scheme which synchronously conducts feature grouping with representation learning. In this way, SMoG solves the problem of supervisory signal hysteresis which the clustering-based method usually faces, and reduces the false negatives of instance contrastive methods. We conduct exhaustive experiments to show that SMoG works well on both CNN and Transformer backbones. Results prove that SMoG has surpassed the current SOTA unsupervised representation learning methods. Moreover, its linear evaluation results surpass the performances obtained by vanilla supervised learning and the representation can be well transferred to downstream tasks.