# Cluster and Predict Latent Patches for Improved Masked Image Modeling

arXiv: https://arxiv.org/abs/2502.08769 · HTML: https://mlsanity.com/paper/2502.08769/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 71 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)

## Built on by (most cited first, 17 of 17)

- [Can Masked Autoencoders Also Listen to Birds?](https://mlsanity.com/paper/2504.12880/) (2025)
- [Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion](https://mlsanity.com/paper/2507.06230/) (2025)
- [In Pursuit of Pixel Supervision for Visual Pre-training](https://mlsanity.com/paper/2512.15715/) (2025)
- [Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification](https://mlsanity.com/paper/2509.24901/) (2025)
- [MuM: Multi-View Masked Image Modeling for 3D Vision](https://mlsanity.com/paper/2511.17309/) (2025)
- [You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences](https://mlsanity.com/paper/2606.15956/) (2026)
- [T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2510.23484/) (2025)
- [Learn from your own latents and not from tokens: A sample-complexity theory](https://mlsanity.com/paper/2605.27734/) (2026)
- [Learning Sparse Visual Representations via Spatial-Semantic Factorization](https://mlsanity.com/paper/2602.01905/) (2026)
- [Separating Representation from Reconstruction Enables Scalable Text Encoders](https://mlsanity.com/paper/2607.04011/) (2026)
- [RoMedFormer: A Rotary-Embedding Transformer Foundation Model for 3D Genito-Pelvic Structure Segmentation in MRI and CT](https://mlsanity.com/paper/2503.14304/) (2025)
- [Collapse of Patches: Ranking Image Patches for Efficient Visual Modeling](https://mlsanity.com/paper/2511.22281/) (2025)
- [Self-Supervised Learning with a Multi-Task Latent Space Objective](https://mlsanity.com/paper/2602.05845/) (2026)
- [BAT: Better Audio Transformer Guided by Convex Gated Probing](https://mlsanity.com/paper/2602.16305/) (2026)
- [AEGIS: A Multi-Task Joint-Embedding Predictive Architecture for Mammography](https://mlsanity.com/paper/2607.00277/) (2026)
- [The JEPA Predictor: A Transferable Operator for Occluded Feature Completion](https://mlsanity.com/paper/2607.16274/) (2026)
- [Emergent Multi-View Geometry Through Self-Distillation](https://mlsanity.com/paper/2609.39227/) (2026)

## Abstract (arXiv)

Masked Image Modeling (MIM) offers a promising approach to self-supervised representation learning, however existing MIM models still lag behind the state-of-the-art. In this paper, we systematically analyze target representations, loss functions, and architectures, to introduce CAPI - a novel pure-MIM framework that relies on the prediction of latent clusterings. Our approach leverages a clustering-based loss, which is stable to train, and exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8% accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes, substantially outperforming previous MIM methods and approaching the performance of the current state-of-the-art, DINOv2. We release all our code and models.