mlsanityData to 14 Oct 2026

VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix

arXiv 2206.08919 ↗First posted 17 Jun 2022cs.CVcited by 45

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Existing vision-language pre-training (VLP) methods primarily rely on paired image-text datasets, which are either annotated by enormous human labors, or crawled from the internet followed by elaborate data cleaning techniques. To reduce the dependency on well-aligned image-text pairs, it is promising to directly leverage the large-scale text-only and image-only corpora. This paper proposes a data augmentation method, namely cross-modal CutMix (CMC), for implicit cross-modal alignment learning in unpaired VLP. Specifically, CMC transforms natural sentences from the textual view into a multi-modal view, where visually-grounded words in a sentence are randomly replaced by diverse image patches with similar semantics. There are several appealing proprieties of the proposed CMC. First, it enhances the data diversity while keeping the semantic meaning intact for tackling problems where the aligned data are scarce; Second, by attaching cross-modal noise on uni-modal data, it guides models to learn token-level interactions across modalities for better denoising. Furthermore, we present a new unpaired VLP method, dubbed as VLMixer, that integrates CMC with contrastive learning to pull together the uni-modal and multi-modal views for better instance-level alignments among different modalities. Extensive experiments on five downstream tasks show that VLMixer could surpass previous state-of-the-art unpaired VLP methods.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 43 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

33 of 43 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
21 more
  1. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  9. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  10. 2017Dense-Captioning Events in Videoscited by 793
  11. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  12. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  13. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  14. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  15. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  16. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  17. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  18. 2019Unpaired Image Captioning via Scene Graph Alignmentscited by 82
  19. 2018Unsupervised Image Captioningcited by 79
  20. 2020VIVO: Visual Vocabulary Pre-Training for Novel Object Captioningcited by 30
  21. 2018Unpaired Image Captioning by Language Pivotingcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 16; 2024: 11; 2025: 12; 2026: 4.

45 papers.

Sort:

By area:

  1. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 156
  2. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  3. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 108
  4. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  5. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  6. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  7. 2023CLIP-guided Prototype Modulating for Few-shot Action Recognitioncited by 41
  8. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectorycited by 36
  9. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 31
  10. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Modelscited by 30
  11. 2023Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualizationcited by 25
  12. 2024A Survey on Mixup Augmentations and Beyondcited by 20
33 more
  1. 2024ProtChatGPT: Towards Understanding Proteins with Large Language Modelscited by 18
  2. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attackcited by 18
  3. 2023Text-based Person Search without Parallel Image-Text Datacited by 17
  4. 2026Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restorationcited by 15
  5. 2023MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognitioncited by 14
  6. 2023RecExplainer: Aligning Large Language Models for Explaining Recommendation Modelscited by 14
  7. 2024Causal Representation Learning from Multimodal Biomedical Observationscited by 13
  8. 2023GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methodscited by 11
  9. 2023Accelerating Vision-Language Pretraining with Free Language Modelingcited by 9
  10. 2024Hierarchical Memory for Long Video QAcited by 9
  11. 2025Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representationscited by 9
  12. 2023Weakly Supervised Vision-and-Language Pre-training with Relative Representationscited by 5
  13. 2024Enhancing Cross-Modal Fine-Tuning with Gradually Intermediate Modality Generationcited by 5
  14. 2024Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representationscited by 5
  15. 2025Open-set Cross Modal Generalization via Multimodal Unified Representationcited by 4
  16. 2023PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Promptingcited by 3
  17. 2025Beyond CLIP Generalization: Against Forward&Backward Forgetting Adapter for Continual Learning of Vision-Language Modelscited by 3
  18. 2025BaryIR: Learning Multi-Source Unified Representation in Continuous Barycenter Space for Generalizable All-in-One Image Restorationcited by 3
  19. 2024Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Modelscited by 2
  20. 2025Diversity Covariance-Aware Prompt Learning for Vision-Language Modelscited by 2
  21. 2025MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learningcited by 2
  22. 2025Breaking the Modality Wall: Time-step Mixup for Efficient Spiking Knowledge Transfer from Static to Event Domaincited by 2
  23. 2023PowMix: A Versatile Regularizer for Multimodal Sentiment Analysiscited by 1
  24. 2024Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Modelscited by 0
  25. 2024Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Modelscited by 0
  26. 2025Continual Cross-Modal Generalizationcited by 0
  27. 2025PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traitscited by 0
  28. 2025Time-step Mixup for Efficient Spiking Knowledge Transfer from Appearance to Event Domaincited by 0
  29. 2025MS-Mix: Sentiment-Guided Adaptive Augmentation for Multimodal Sentiment Analysiscited by 0
  30. 2025Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attackcited by 0
  31. 2026Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalizationcited by 0
  32. 2026Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representationscited by 0
  33. 2026Generative Retrieval for Unsupervised Text-Based Person Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training
  2. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
  3. 2022UNIMO-2: End-to-End Unified Vision-Language Grounded Learning
  4. 2024NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
  5. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
  6. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate
  7. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddings
  8. 2022Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learning