mlsanityData to 7 Oct 2026

SemanticMIM: Marring Masked Image Modeling with Semantics Compression for General Visual Representation

arXiv 2406.10673 ↗First posted 15 Jun 2024cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

This paper represents a neat yet effective framework, named SemanticMIM, to integrate the advantages of masked image modeling (MIM) and contrastive learning (CL) for general visual representation. We conduct a thorough comparative analysis between CL and MIM, revealing that their complementary advantages fundamentally stem from two distinct phases, i.e., compression and reconstruction. Specifically, SemanticMIM leverages a proxy architecture that customizes interaction between image and mask tokens, bridging these two phases to achieve general visual representation with the property of abundant semantic and positional awareness. Through extensive qualitative and quantitative evaluations, we demonstrate that SemanticMIM effectively amalgamates the benefits of CL and MIM, leading to significant enhancement of performance and feature linear separability. SemanticMIM also offers notable interpretability through attention response visualization. Codes are available at https://github.com/yyk-wew/SemanticMIM.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 66 of its 70 references that are in the map. First posted on arXiv 15 Jun 2024.

Builds on

66 of 70 references.

References · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  10. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  13. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  14. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  15. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  16. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  17. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  18. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  19. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  20. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  21. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  22. 2023A Cookbook of Self-Supervised Learningcited by 248
  23. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  24. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  25. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  26. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  27. 2020Unsupervised Learning of Dense Visual Representationscited by 147
  28. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  29. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 117
  30. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  31. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  32. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  33. 2021Instance Localization for Self-supervised Detection Pretrainingcited by 99
  34. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  35. 2022Self-Supervised Learning of Object Parts for Semantic Segmentationcited by 90
  36. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  37. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  38. 2021Unsupervised Object-Level Representation Learning from Scene Imagescited by 72

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Beyond [cls]: Exploring the true potential of Masked Image Modeling representationscited by 6
  2. 2024CLASH: Complementary Learning with Neural Architecture Search for Gait Recognitioncited by 3