mlsanityData to 7 Oct 2026

Masked Vision and Language Modeling for Multi-modal Representation Learning

arXiv 2208.02131 ↗First posted 3 Aug 2022cs.CV cs.CL cs.LGcited by 66

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we study how to use masked signal modeling in vision and language (V+L) representation learning. Instead of developing masked language modeling (MLM) and masked image modeling (MIM) independently, we propose to build joint masked vision and language modeling, where the masked signal of one modality is reconstructed with the help from another modality. This is motivated by the nature of image-text paired data that both of the image and the text convey almost the same information but in different formats. The masked signal reconstruction of one modality conditioned on another modality can also implicitly learn cross-modal alignment between language tokens and image patches. Our experiments on various V+L tasks show that the proposed method, along with common V+L alignment losses, achieves state-of-the-art performance in the regime of millions of pre-training data. Also, we outperforms the other competitors by a significant margin in limited data scenarios.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 54 of its 76 references that are in the map. First posted on arXiv 3 Aug 2022.

Builds on

54 of 76 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2016Layer Normalizationcited by 5,579
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  13. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  14. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  15. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  16. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  17. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  18. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  19. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  20. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  21. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  22. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  23. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  24. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  25. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  26. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  27. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  28. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  29. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  30. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  31. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  32. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  33. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  34. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  35. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  36. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  37. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  38. 2013Generalized Denoising Auto-Encoders as Generative Modelscited by 113

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 17; 2024: 23; 2025: 15; 2026: 6.

66 papers.

Sort:

By area:

  1. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  2. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  3. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  4. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  6. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  7. 2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
  8. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 68
  9. 2023Hard Patches Mining for Masked Image Modelingcited by 56
  10. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  11. 2023Mask Grounding for Referring Image Segmentationcited by 40
  12. 2024Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlookcited by 33
38 more
  1. 2023FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Spacecited by 31
  2. 2024Masked Image Modeling: A Surveycited by 28
  3. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 24
  4. 2023Hulk: A Universal Knowledge Translator for Human-Centric Taskscited by 22
  5. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  6. 2024Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognitioncited by 18
  7. 2025SARLANG-1M: A Benchmark for Vision-Language Modeling in SAR Image Understandingcited by 16
  8. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  9. 2024Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencodercited by 15
  10. 2022SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-trainingcited by 14
  11. 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14
  12. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  13. 2023Global and Local Semantic Completion Learning for Vision-Language Pre-trainingcited by 12
  14. 2024Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensingcited by 12
  15. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  16. 2022MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learningcited by 11
  17. 2024SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretrainingcited by 8
  18. 2024PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representationcited by 8
  19. 2023Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detectioncited by 7
  20. 2024MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuningcited by 6
  21. 2024PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal Documentscited by 6
  22. 2024Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptionscited by 6
  23. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Surveycited by 6
  24. 2024Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinementcited by 5
  25. 2024IntCoOp: Interpretability-Aware Vision-Language Prompt Tuningcited by 5
  26. 2025ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Expertscited by 5
  27. 2024From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signalscited by 4
  28. 2025DGTRSD & DGTRS-CLIP: A Dual-Granularity Remote Sensing Image-Text Dataset and Vision Language Foundation Model for Alignmentcited by 4
  29. 2025REJEPA: A Novel Joint-Embedding Predictive Architecture for Efficient Remote Sensing Image Retrievalcited by 4
  30. 2025EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognitioncited by 4
  31. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  32. 2024Unified Multi-modal Diagnostic Framework with Reconstruction Pre-training and Heterogeneity-combat Tuningcited by 3
  33. 2024Masks and Manuscripts: Advancing Medical Pre-training with End-to-End Masking and Narrative Structuringcited by 3
  34. 2024TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Modelscited by 3
  35. 2025Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completioncited by 2
  36. 2025Neural Scene Designer: Self-Styled Semantic Image Manipulationcited by 2
  37. 2024NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionalitycited by 1
  38. 2024CAST: Cross-modal Alignment Similarity Test for Vision Language Modelscited by 1

50 most cited of 66 All 66 (JSON)