mlsanityData to 7 Oct 2026

Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training

arXiv 2412.16148 ↗First posted 20 Dec 2024cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision Language Models (VLMs) can be trained more efficiently if training sets can be reduced in size. Recent work has shown the benefits of masking text during VLM training using a variety of strategies (truncation, random masking, block masking and syntax masking) and has reported syntax masking as the top performer. In this paper, we analyze the impact of different text masking strategies on the word frequency in the training data, and show that this impact is connected to model success. This finding motivates Contrastive Language-Image Pre-training with Word Frequency Masking (CLIPF), our proposed masking approach, which directly leverages word frequency. Extensive experiments demonstrate the advantages of CLIPF over syntax masking and other existing approaches, particularly when the number of input tokens decreases. We show that not only CLIPF, but also other existing masking strategies, outperform syntax masking when enough epochs are used during training, a finding of practical importance for selecting a text masking method for VLM training. Our code is available online.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 47 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

32 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
19 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2019Natural Adversarial Examplescited by 1,401
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  9. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  10. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  11. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  12. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  13. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  14. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  15. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  16. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  17. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  18. 2022Attentive Mask CLIPcited by 23
  19. 2023RECLIP: Resource-efficient CLIP by Training with Small Imagescited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Uniform Masking Prevails in Vision-Language Pretraining
  2. 2024Enhancing Vision-Language Model Pre-training with Image-text Pair Pruning Based on Word Frequency
  3. 2021Data Efficient Masked Language Modeling for Vision and Language
  4. 2024Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification
  5. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  6. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  7. 2024Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models
  8. 2026Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation