mlsanityData to 7 Oct 2026

E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning

arXiv 2106.01804 ↗First posted 3 Jun 2021cs.CV cs.AI cs.CLcited by 84

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language pre-training (VLP) on large-scale image-text pairs has achieved huge success for the cross-modal downstream tasks. The most existing pre-training methods mainly adopt a two-step training procedure, which firstly employs a pre-trained object detector to extract region-based visual features, then concatenates the image representation and text embedding as the input of Transformer to train. However, these methods face problems of using task-specific visual representation of the specific object detector for generic cross-modal understanding, and the computation inefficiency of two-stage pipeline. In this paper, we propose the first end-to-end vision-language pre-trained model for both V+L understanding and generation, namely E2E-VLP, where we build a unified Transformer framework to jointly learn visual representation, and semantic alignments between image and text. We incorporate the tasks of object detection and image captioning into pre-training with a unified Transformer encoder-decoder architecture for enhancing visual learning. An extensive set of experiments have been conducted on well-established vision-language downstream tasks to demonstrate the effectiveness of this novel VLP paradigm.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 42 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

38 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels14 Mar

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
25 more
  1. 2015VQA: Visual Question Answeringcited by 2,489
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2018Image Transformercited by 770
  11. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  12. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  13. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  14. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  15. 2020Circle Loss: A Unified Perspective of Pair Similarity Optimizationcited by 402
  16. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  17. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  18. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  19. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  20. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  21. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  22. 2015End-to-end people detection in crowded scenescited by 140
  23. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  24. 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 108
  25. 2020PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generationcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 35; 2023: 21; 2024: 14; 2025: 3; 2026: 1.

84 papers.

Sort:

By area:

  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  3. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  4. 2022Multimodal Learning with Transformers: A Surveycited by 399
  5. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  6. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  7. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  8. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  9. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  10. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  11. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  12. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
38 more
  1. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  2. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  3. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  4. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  5. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  6. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  7. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  8. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  9. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  10. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  11. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  12. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  13. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  14. 2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
  15. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Modelcited by 41
  16. 2023Transformers in Speech Processing: A Surveycited by 38
  17. 2021EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matchingcited by 36
  18. 2022FashionViL: Fashion-Focused Vision-and-Language Representation Learningcited by 32
  19. 2022Named Entity and Relation Extraction with Multi-Modal Retrievalcited by 26
  20. 2023InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluationcited by 25
  21. 2021KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillationcited by 23
  22. 2021Distilled Dual-Encoder Model for Vision-Language Understandingcited by 21
  23. 2022UNIMO-2: End-to-End Unified Vision-Language Grounded Learningcited by 19
  24. 2023TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selectioncited by 18
  25. 2021Towards Language-guided Visual Recognition via Dynamic Convolutionscited by 17
  26. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learnerscited by 17
  27. 2022Controllable Image Captioning via Promptingcited by 15
  28. 2023Transforming Visual Scene Graphs to Image Captionscited by 15
  29. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reductioncited by 15
  30. 2022Masked Vision-Language Transformer in Fashioncited by 14
  31. 2022Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learningcited by 14
  32. 2022Efficient Image Captioning for Edge Devicescited by 14
  33. 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14
  34. 2024From Data Deluge to Data Curation: A Filtering-WoRA Paradigm for Efficient Text-based Person Searchcited by 14
  35. 2022MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learningcited by 13
  36. 2022FindIt: Generalized Localization with Natural Language Queriescited by 13
  37. 2023ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representationcited by 13
  38. 2022A Frustratingly Simple Approach for End-to-End Image Captioningcited by 12

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  2. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
  3. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
  4. 2021Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  6. 2022A Survey of Vision-Language Pre-Trained Models
  7. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
  8. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation