mlsanityData to 14 Oct 2026

Accelerating Vision-Language Pretraining with Free Language Modeling

arXiv 2303.14038 ↗First posted 24 Mar 2023cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The state of the arts in vision-language pretraining (VLP) achieves exemplary performance but suffers from high training costs resulting from slow convergence and long training time, especially on large-scale web datasets. An essential obstacle to training efficiency lies in the entangled prediction rate (percentage of tokens for reconstruction) and corruption rate (percentage of corrupted tokens) in masked language modeling (MLM), that is, a proper corruption rate is achieved at the cost of a large portion of output tokens being excluded from prediction loss. To accelerate the convergence of VLP, we propose a new pretraining task, namely, free language modeling (FLM), that enables a 100% prediction rate with arbitrary corruption rates. FLM successfully frees the prediction rate from the tie-up with the corruption rate while allowing the corruption spans to be customized for each token to be predicted. FLM-trained models are encouraged to learn better and faster given the same GPU time by exploiting bidirectional contexts more flexibly. Extensive experiments show FLM could achieve an impressive 2.5x pretraining time reduction in comparison to the MLM-based methods, while keeping competitive performance on both vision-language understanding and generation tasks. Code will be public at https://github.com/TencentARC/FLM.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 45 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

38 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix17 Jun
  2. 2022VL-BEiT: Generative Vision-Language Pretraining2 Jun
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  4. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  5. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov
  6. 2021Data Efficient Masked Language Modeling for Vision and Language5 Sep
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
19 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  9. 2021Grounded Language-Image Pre-trainingcited by 1,156
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  11. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  12. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  13. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  14. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  15. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  16. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  17. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  18. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  19. 2022GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-trainingcited by 10

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 5; 2025: 1.

9 papers.

  1. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 108
  2. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99
  3. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  4. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectorycited by 36
  5. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attackcited by 18
  6. 2024Text-Video Retrieval with Global-Local Semantic Consistent Learningcited by 3
  7. 2024Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-trainingcited by 1
  8. 2024Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Modelscited by 0
  9. 2025Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Uniform Masking Prevails in Vision-Language Pretraining
  2. 2022Leveraging per Image-Token Consistency for Vision-Language Pre-training
  3. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
  4. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction
  5. 2023Effective End-to-End Vision Language Pretraining with Semantic Visual Loss
  6. 2024POINTS: Improving Your Vision-language Model with Affordable Strategies
  7. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
  8. 2022Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics