mlsanityData to 7 Oct 2026

SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

arXiv 2103.07829 ↗First posted 14 Mar 2021cs.CLcited by 19

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language pre-training (VLP) on large-scale image-text pairs has recently witnessed rapid progress for learning cross-modal representations. Existing pre-training methods either directly concatenate image representation and text representation at a feature level as input to a single-stream Transformer, or use a two-stream cross-modal Transformer to align the image-text representation at a high-level semantic space. In real-world image-text data, we observe that it is easy for some of the image-text pairs to align simple semantics on both modalities, while others may be related after higher-level abstraction. Therefore, in this paper, we propose a new pre-training method SemVLP, which jointly aligns both the low-level and high-level semantics between image and text representations. The model is pre-trained iteratively with two prevalent fashions: single-stream pre-training to align at a fine-grained feature level and two-stream pre-training to align high-level semantics, by employing a shared Transformer network with a pluggable cross-modal attention module. An extensive set of experiments have been conducted on four well-established vision-language understanding tasks to demonstrate the effectiveness of the proposed SemVLP in aligning cross-modal representations towards different semantic granularities.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 28 references that are in the map. First posted on arXiv 14 Mar 2021.

Builds on

21 of 28 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
9 more
  1. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  2. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  3. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  4. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  5. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  6. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  7. 2019Learning by Abstraction: The Neural State Machinecited by 113
  8. 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 108
  9. 2019Meta Module Network for Compositional Visual Reasoningcited by 39

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 6; 2023: 4; 2024: 2; 2025: 1.

19 papers.

Sort:

By area:

  1. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  4. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84
  5. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  6. 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignmentcited by 19
  7. 2021Data Efficient Masked Language Modeling for Vision and Languagecited by 18
  8. 2021CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classificationcited by 16
  9. 2022Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrasescited by 16
  10. 2021What Vision-Language Models `See' when they See Scenescited by 8
  11. 2023Missing-modality Enabled Multi-modal Fusion Architecture for Medical Datacited by 8
  12. 2022Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answeringcited by 7
7 more
  1. 2023Masked Autoencoding Does Not Help Natural Language Supervision at Scalecited by 7
  2. 2023UniMAP: Universal SMILES-Graph Representation Learningcited by 6
  3. 2021Achieving Human Parity on Visual Question Answeringcited by 5
  4. 2024One-Stage Open-Vocabulary Temporal Action Detection Leveraging Temporal Multi-scale and Action Label Featurescited by 2
  5. 2022Modeling Paragraph-Level Vision-Language Semantic Alignment for Multi-Modal Summarizationcited by 1
  6. 2024CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Trainingcited by 0
  7. 2025Improving vision-language alignment with graph spiking hybrid Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
  2. 2024Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training
  3. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  4. 2024NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
  5. 2022UNIMO-2: End-to-End Unified Vision-Language Grounded Learning
  6. 2022MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning
  7. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
  8. 2022A Survey of Vision-Language Pre-Trained Models