# Mugs: A Multi-Granular Self-Supervised Learning Framework

arXiv: https://arxiv.org/abs/2203.14415 · HTML: https://mlsanity.com/paper/2203.14415/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 56 references that are in the map.
- First posted on arXiv 27 Mar 2022.

## Builds on (most cited first, 30 of 46)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)

## Built on by (most cited first, 30 of 54)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models](https://mlsanity.com/paper/2208.06677/) (2022)
- [RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank](https://mlsanity.com/paper/2210.02885/) (2022)
- [VICRegL: Self-Supervised Learning of Local Visual Features](https://mlsanity.com/paper/2210.01571/) (2022)
- [On the duality between contrastive and non-contrastive self-supervised learning](https://mlsanity.com/paper/2206.02574/) (2022)
- [MILAN: Masked Image Pretraining on Language Assisted Representation](https://mlsanity.com/paper/2208.06049/) (2022)
- [LPT: Long-tailed Prompt Tuning for Image Classification](https://mlsanity.com/paper/2210.01033/) (2022)
- [Diffusion Models Beat GANs on Image Classification](https://mlsanity.com/paper/2307.08702/) (2023)
- [Self-supervised learning of Split Invariant Equivariant representations](https://mlsanity.com/paper/2302.10283/) (2023)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [Do text-free diffusion models learn discriminative visual representations?](https://mlsanity.com/paper/2311.17921/) (2023)
- [Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning](https://mlsanity.com/paper/2206.13378/) (2022)
- [Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works](https://mlsanity.com/paper/2211.08129/) (2022)
- [Learning high-level visual representations from a child's perspective without strong inductive biases](https://mlsanity.com/paper/2305.15372/) (2023)
- [Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning](https://mlsanity.com/paper/2507.14137/) (2025)
- [MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representations](https://mlsanity.com/paper/2402.10093/) (2024)
- [Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks](https://mlsanity.com/paper/2206.03826/) (2022)
- [Attention Distillation: self-supervised vision transformer students need more guidance](https://mlsanity.com/paper/2210.00944/) (2022)
- [Neural Eigenfunctions Are Structured Representation Learners](https://mlsanity.com/paper/2210.12637/) (2022)
- [What Variables Affect Out-of-Distribution Generalization in Pretrained Models?](https://mlsanity.com/paper/2405.15018/) (2024)
- [SERE: Exploring Feature Self-relation for Self-supervised Transformer](https://mlsanity.com/paper/2206.05184/) (2022)
- [No More Shortcuts: Realizing the Potential of Temporal Self-Supervision](https://mlsanity.com/paper/2312.13008/) (2023)
- [Towards Sustainable Self-supervised Learning](https://mlsanity.com/paper/2210.11016/) (2022)
- [Contrasting Multiple Representations with the Multi-Marginal Matching Gap](https://mlsanity.com/paper/2405.19532/) (2024)
- [Visual DNA: Representing and Comparing Images using Distributions of Neuron Activations](https://mlsanity.com/paper/2304.10036/) (2023)
- [On Pretraining Data Diversity for Self-Supervised Learning](https://mlsanity.com/paper/2403.13808/) (2024)
- [Understanding the Role of Equivariance in Self-supervised Learning](https://mlsanity.com/paper/2411.06508/) (2024)
- [Evaluating Self-Supervised Learning via Risk Decomposition](https://mlsanity.com/paper/2302.03068/) (2023)
- [In-Context Symmetries: Self-Supervised Learning through Contextual World Models](https://mlsanity.com/paper/2405.18193/) (2024)

## Abstract (arXiv)

In self-supervised learning, multi-granular features are heavily desired though rarely investigated, as different downstream tasks (e.g., general and fine-grained classification) often require different or multi-granular features, e.g.~fine- or coarse-grained one or their mixture. In this work, for the first time, we propose an effective MUlti-Granular Self-supervised learning (Mugs) framework to explicitly learn multi-granular visual features. Mugs has three complementary granular supervisions: 1) an instance discrimination supervision (IDS), 2) a novel local-group discrimination supervision (LGDS), and 3) a group discrimination supervision (GDS). IDS distinguishes different instances to learn instance-level fine-grained features. LGDS aggregates features of an image and its neighbors into a local-group feature, and pulls local-group features from different crops of the same image together and push them away for others. It provides complementary instance supervision to IDS via an extra alignment on local neighbors, and scatters different local-groups separately to increase discriminability. Accordingly, it helps learn high-level fine-grained features at a local-group level. Finally, to prevent similar local-groups from being scattered randomly or far away, GDS brings similar samples close and thus pulls similar local-groups together, capturing coarse-grained features at a (semantic) group level. Consequently, Mugs can capture three granular features that often enjoy higher generality on diverse downstream tasks over single-granular features, e.g.~instance-level fine-grained features in contrastive learning. By only pretraining on ImageNet-1K, Mugs sets new SoTA linear probing accuracy 82.1% on ImageNet-1K and improves previous SoTA by 1.1%. It also surpasses SoTAs on other tasks, e.g. transfer learning, detection and segmentation.