mlsanityData to 7 Oct 2026

MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representations

arXiv 2402.10093 ↗First posted 15 Feb 2024cs.CV cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

We introduce MIM (Masked Image Modeling)-Refiner, a contrastive learning boost for pre-trained MIM models. MIM-Refiner is motivated by the insight that strong representations within MIM models generally reside in intermediate layers. Accordingly, MIM-Refiner leverages multiple contrastive heads that are connected to different intermediate layers. In each head, a modified nearest neighbor objective constructs semantic clusters that capture semantic information which improves performance on downstream tasks, including off-the-shelf and fine-tuning settings. The refinement process is short and simple - yet highly effective. Within a few epochs, we refine the features of MIM models from subpar to state-of-the-art, off-the-shelf features. Refining a ViT-H, pre-trained with data2vec 2.0 on ImageNet-1K, sets a new state-of-the-art in linear probing (84.7%) and low-shot classification among models that are pre-trained on ImageNet-1K. MIM-Refiner efficiently combines the advantages of MIM and ID objectives and compares favorably against previous state-of-the-art SSL models on a variety of benchmarks such as low-shot classification, long-tailed classification, clustering and semantic segmentation.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 72 of its 104 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

72 of 104 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2023Segment Anythingcited by 8,158
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  13. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  14. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  15. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  16. 2013Describing Textures in the Wildcited by 2,110
  17. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  18. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  19. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  20. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  21. 2019Natural Adversarial Examplescited by 1,401
  22. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  23. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  24. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  25. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  26. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  27. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  28. 2016Deep Networks with Stochastic Depthcited by 973
  29. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  30. 2021Going deeper with Image Transformerscited by 759
  31. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  32. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  33. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  34. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  35. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  36. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  37. 2014Deeply-Supervised Netscited by 364
  38. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 8; 2026: 5.

19 papers.

Sort:

By area:

  1. 2025REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjectscited by 70
  2. 2024Vision-LSTM: xLSTM as Generic Vision Backbonecited by 51
  3. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  4. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  5. 2025Cluster and Predict Latent Patches for Improved Masked Image Modelingcited by 17
  6. 2024Let Go of Your Labels with Unsupervised Transfercited by 14
  7. 2024ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopycited by 13
  8. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 12
  9. 2025Pre-trained Forecasting Models: Strong Zero-Shot Feature Extractors for Time Series Classificationcited by 9
  10. 2025Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classificationcited by 7
  11. 2024Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation modelscited by 4
  12. 2024Scaling Up Deep Clustering Methods Beyond ImageNet-1Kcited by 1
7 more
  1. 2026Multiview Self-Representation Learning across Heterogeneous Viewscited by 1
  2. 2024When Mamba Meets xLSTM: An Efficient and Precise Method with the xLSTM-VMUNet Model for Skin lesion Segmentationcited by 0
  3. 2025Unsupervised Image Classification with Adaptive Nearest Neighbor Selection and Cluster Ensemblescited by 0
  4. 2026Suppressing Non-Semantic Noise in Masked Image Modeling Representationscited by 0
  5. 2026Learning Subspace-Preserving Sparse Attention Graphs from Heterogeneous Multiview Datacited by 0
  6. 2026Effective Biological Representation Learning by Masking Gene Expressioncited by 0
  7. 2026MABLE: Masked Autoencoding with Bi-Lipschitz Decoding for Embeddings and Graph Metric Learningcited by 0