mlsanityData to 14 Oct 2026

Efficient Vision-Language Pre-training by Cluster Masking

arXiv 2405.08815 ↗First posted 14 May 2024cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We propose a simple strategy for masking image patches during visual-language contrastive learning that improves the quality of the learned representations and the training speed. During each iteration of training, we randomly mask clusters of visually similar image patches, as measured by their raw pixel intensities. This provides an extra learning signal, beyond the contrastive training itself, since it forces a model to predict words for masked visual structures solely from context. It also speeds up training by reducing the amount of data used in each image. We evaluate the effectiveness of our model by pre-training on a number of benchmarks, finding that it outperforms other masking strategies, such as FLIP, on the quality of the learned representation.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 69 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

49 of 69 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
36 more
  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2019Natural Adversarial Examplescited by 1,408
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  13. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  14. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  15. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  16. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  17. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  18. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  19. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  20. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  21. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  22. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  23. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  24. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  25. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  26. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  27. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  28. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
  29. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  30. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 114
  31. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  32. 2022Adversarial Masking for Self-Supervised Learningcited by 78
  33. 2022Masked Frequency Modeling for Self-Supervised Visual Pre-Trainingcited by 71
  34. 2023SwinMM: Masked Multi-view with Swin Transformers for 3D Medical Image Segmentationcited by 35
  35. 2022Out of Distribution Detection on ImageNet-Ocited by 5
  36. 2023Difference-Masking: Choosing What to Mask in Continued Pretrainingcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 10.

13 papers.

Sort:

By area:

  1. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Typescited by 10
  2. 2025Seeing What Matters: Empowering CLIP with Patch Generation-to-Selectioncited by 7
  3. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memescited by 4
  4. 2025Random Registers for Cross-Domain Few-Shot Learningcited by 4
  5. 2025SuperCLIP: CLIP with Simple Classification Supervisioncited by 4
  6. 2025Learning Event Completeness for Weakly Supervised Video Anomaly Detectioncited by 3
  7. 2025DiffCLIP: Differential Attention Meets CLIPcited by 2
  8. 2025Multimodal Representation Learning Conditioned on Semantic Relationscited by 1
  9. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Trainingcited by 1
  10. 2024Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selectioncited by 0
  11. 2025Progressive Local Alignment for Medical Multimodal Pre-trainingcited by 0
  12. 2025Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selectioncited by 0
1 more
  1. 2025Cross-Domain Few-Shot Learning via Multi-View Collaborative Optimization with Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
  2. 2022Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics
  3. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representations
  4. 2022Uniform Masking Prevails in Vision-Language Pretraining
  5. 2022Leveraging per Image-Token Consistency for Vision-Language Pre-training
  6. 2021Data Efficient Masked Language Modeling for Vision and Language
  7. 2024Adaptive Masking Enhances Visual Grounding
  8. 2024Centered Masking for Language-Image Pre-Training