Uniform Masking Prevails in Vision-Language Pretraining
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Masked Language Modeling (MLM) has proven to be an essential component of Vision-Language (VL) pretraining. To implement MLM, the researcher must make two design choices: the masking strategy, which determines which tokens to mask, and the masking rate, which determines how many tokens to mask. Previous work has focused primarily on the masking strategy while setting the masking rate at a default of 15%. In this paper, we show that increasing this masking rate improves downstream performance while simultaneously reducing performance gap among different masking strategies, rendering the uniform masking strategy competitive to other more complex ones. Surprisingly, we also discover that increasing the masking rate leads to gains in Image-Text Matching (ITM) tasks, suggesting that the role of MLM goes beyond language modeling in VL pretraining.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 29 references that are in the map. First posted on arXiv 10 Dec 2022.
Builds on
21 of 29 references.
Closest ancestors · 1
Same direction, newest first
Other references · 20
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
8 more
- 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
- 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
- 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
- 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
- 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
- 2022Should You Mask 15% in Masked Language Modeling?cited by 118
- 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 109
- 2020PMI-Masking: Principled masking of correlated spanscited by 54
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2023Accelerating Vision-Language Pretraining with Free Language Modeling
- 2022Leveraging per Image-Token Consistency for Vision-Language Pre-training
- 2022What to Hide from Your Students: Attention-Guided Masked Image Modeling
- 2024Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training
- 2024Efficient Vision-Language Pre-training by Cluster Masking
- 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
- 2022Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics
- 2023VILA: On Pre-training for Visual Language Models