mlsanityData to 7 Oct 2026

Data Efficient Masked Language Modeling for Vision and Language

arXiv 2109.02040 ↗First posted 5 Sep 2021cs.CL cs.CV cs.LGcited by 18

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Masked language modeling (MLM) is one of the key sub-tasks in vision-language pretraining. In the cross-modal setting, tokens in the sentence are masked at random, and the model predicts the masked tokens given the image and the text. In this paper, we observe several key disadvantages of MLM in this setting. First, as captions tend to be short, in a third of the sentences no token is sampled. Second, the majority of masked tokens are stop-words and punctuation, leading to under-utilization of the image. We investigate a range of alternative masking strategies specific to the cross-modal setting that address these shortcomings, aiming for better fusion of text and image in the learned representation. When pre-training the LXMERT model, our alternative masking strategies consistently improve over the original masking strategy on three downstream tasks, especially in low resource settings. Further, our pre-training approach substantially outperforms the baseline model on a prompt-based probing task designed to elicit image objects. These results and our analysis indicate that our method allows for better utilization of the training data.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 40 references that are in the map. First posted on arXiv 5 Sep 2021.

Builds on

29 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels14 Mar
  2. 2021Causal Attention for Vision-Language Tasks5 Mar

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  12. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
15 more
  1. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  2. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  3. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 301
  4. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  5. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  6. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  7. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  8. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  9. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
  10. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
  11. 2020PMI-Masking: Principled masking of correlated spanscited by 54
  12. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  13. 2020Object-Centric Diagnosis of Visual Reasoningcited by 16
  14. 2021Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Visioncited by 14
  15. 2021Answer Questions with Right Image Regions: A Visual Attention Regularization Approachcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 8; 2023: 7; 2024: 1; 2025: 1.

18 papers.

Sort:

By area:

  1. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  2. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  3. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  4. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  5. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  6. 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
  7. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19
  8. 2023Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understandingcited by 19
  9. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  10. 2022GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-trainingcited by 10
  11. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  12. 2022Leveraging per Image-Token Consistency for Vision-Language Pre-trainingcited by 9
6 more
  1. 2023Accelerating Vision-Language Pretraining with Free Language Modelingcited by 9
  2. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  3. 2022Vision-and-Language Pretrainingcited by 2
  4. 2022Training Vision-Language Models with Less Bimodal Supervisioncited by 2
  5. 2023Read, Look or Listen? What's Needed for Solving a Multimodal Datasetcited by 2
  6. 2022Uniform Masking Prevails in Vision-Language Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training
  2. 2023Efficient Masked Autoencoders with Self-Consistency
  3. 2024Efficient Vision-Language Pre-training by Cluster Masking
  4. 2025RIV: Recursive Introspection Mask Diffusion Vision Language Model
  5. 2025ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
  6. 2022Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning
  7. 2022Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics
  8. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Tasks