mlsanityData to 7 Oct 2026

VLMAE: Vision-Language Masked Autoencoder

arXiv 2208.09374 ↗First posted 19 Aug 2022cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Image and language modeling is of crucial importance for vision-language pre-training (VLP), which aims to learn multi-modal representations from large-scale paired image-text data. However, we observe that most existing VLP methods focus on modeling the interactions between image and text features while neglecting the information disparity between image and text, thus suffering from focal bias. To address this problem, we propose a vision-language masked autoencoder framework (VLMAE). VLMAE employs visual generative learning, facilitating the model to acquire fine-grained and unbiased features. Unlike the previous works, VLMAE pays attention to almost all critical patches in an image, providing more comprehensive understanding. Extensive experiments demonstrate that VLMAE achieves better performance in various vision-language downstream tasks, including visual question answering, image-text retrieval and visual grounding, even with up to 20% pre-training speedup.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 39 references that are in the map. First posted on arXiv 19 Aug 2022.

Builds on

32 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb
  2. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  3. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  4. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
16 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  4. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  5. 2016Modeling Context in Referring Expressionscited by 1,054
  6. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  7. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  8. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  9. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  10. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  11. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  12. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  13. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  14. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  15. 2022mc-BEiT: Multi-choice Discretization for Image BERT Pre-trainingcited by 30
  16. 2022Training Vision-Language Transformers from Captionscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 0; 2025: 1; 2026: 2.

11 papers.

  1. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  2. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  3. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 24
  4. 2023Graph Masked Autoencoder for Sequential Recommendationcited by 19
  5. 2022Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learningcited by 14
  6. 2023Global and Local Semantic Completion Learning for Vision-Language Pre-trainingcited by 12
  7. 2025Solving Instance Detection from an Open-World Perspectivecited by 6
  8. 2023Cross-modal Contrastive Learning with Asymmetric Co-attention Network for Video Moment Retrievalcited by 4
  9. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  10. 2026Location-Aware Pretraining for Medical Difference Visual Question Answeringcited by 1
  11. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022VL-BEiT: Generative Vision-Language Pretraining
  2. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
  3. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
  4. 2026Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders
  5. 2023Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment
  6. 2022Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering
  7. 2023MAGVLT: Masked Generative Vision-and-Language Transformer
  8. 2024Probing Visual Language Priors in VLMs