mlsanityData to 14 Oct 2026

Cluster and Predict Latent Patches for Improved Masked Image Modeling

arXiv 2502.08769 ↗First posted 12 Feb 2025cs.CV cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Masked Image Modeling (MIM) offers a promising approach to self-supervised representation learning, however existing MIM models still lag behind the state-of-the-art. In this paper, we systematically analyze target representations, loss functions, and architectures, to introduce CAPI - a novel pure-MIM framework that relies on the prediction of latent clusterings. Our approach leverages a clustering-based loss, which is stable to train, and exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8% accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes, substantially outperforming previous MIM methods and approaching the performance of the current state-of-the-art, DINOv2. We release all our code and models.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 51 of its 71 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

51 of 71 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
38 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  8. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  9. 2019Natural Adversarial Examplescited by 1,408
  10. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  12. 2016Colorful Image Colorizationcited by 1,114
  13. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  14. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  15. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  16. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  17. 2023Vision Transformers Need Registerscited by 871
  18. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  19. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  20. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  21. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  22. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
  23. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  24. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  25. 2023RoMa: Robust Dense Feature Matchingcited by 302
  26. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
  27. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  28. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  29. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  30. 2020Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligencecited by 82
  31. 2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
  32. 2023Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on Aerial Lidarcited by 60
  33. 2024Rethinking Patch Dependence for Masked Autoencoderscited by 29
  34. 2024MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representationscited by 19
  35. 2024Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learningcited by 18
  36. 2024SIGMA: Sinkhorn-Guided Masked Video Modelingcited by 14
  37. 2024T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Datacited by 14
  38. 2023Stochastic positional embeddings improve masked image modelingcited by 12

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 9.

17 papers.

Sort:

By area:

  1. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  2. 2025Feed-Forward SceneDINO for Unsupervised Semantic Scene Completioncited by 13
  3. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 13
  4. 2025Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classificationcited by 7
  5. 2025MuM: Multi-View Masked Image Modeling for 3D Visioncited by 5
  6. 2026You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differencescited by 5
  7. 2025T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learningcited by 2
  8. 2026Learn from your own latents and not from tokens: A sample-complexity theorycited by 2
  9. 2026Learning Sparse Visual Representations via Spatial-Semantic Factorizationcited by 1
  10. 2026Separating Representation from Reconstruction Enables Scalable Text Encoderscited by 1
  11. 2025RoMedFormer: A Rotary-Embedding Transformer Foundation Model for 3D Genito-Pelvic Structure Segmentation in MRI and CTcited by 0
  12. 2025Collapse of Patches: Ranking Image Patches for Efficient Visual Modelingcited by 0
5 more
  1. 2026Self-Supervised Learning with a Multi-Task Latent Space Objectivecited by 0
  2. 2026BAT: Better Audio Transformer Guided by Convex Gated Probingcited by 0
  3. 2026AEGIS: A Multi-Task Joint-Embedding Predictive Architecture for Mammographycited by 0
  4. 2026The JEPA Predictor: A Transferable Operator for Occluded Feature Completioncited by 0
  5. 2026Emergent Multi-View Geometry Through Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
  2. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models
  3. 2026ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modeling
  4. 2023Efficient Masked Autoencoders with Self-Consistency
  5. 2024Membership Inference Attack Against Masked Image Modeling
  6. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation
  7. 2026MEDiC: Multi-objective Exploration of Distillation from CLIP
  8. 2022What to Hide from Your Students: Attention-Guided Masked Image Modeling