mlsanityData to 7 Oct 2026

Effective End-to-End Vision Language Pretraining with Semantic Visual Loss

arXiv 2301.07236 ↗First posted 18 Jan 2023cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Current vision language pretraining models are dominated by methods using region visual features extracted from object detectors. Given their good performance, the extract-then-process pipeline significantly restricts the inference speed and therefore limits their real-world use cases. However, training vision language models from raw image pixels is difficult, as the raw image pixels give much less prior knowledge than region features. In this paper, we systematically study how to leverage auxiliary visual pretraining tasks to help training end-to-end vision language models. We introduce three types of visual losses that enable much faster convergence and better finetuning accuracy. Compared with region feature models, our end-to-end models could achieve similar or better performance on downstream tasks and run more than 10 times faster during inference. Compared with other end-to-end models, our proposed method could achieve similar or better performance when pretrained for only 10% of the pretraining GPU hours.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 50 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
20 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  9. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  12. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  13. 2020Rethinking Pre-training and Self-trainingcited by 352
  14. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  15. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  16. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  17. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  18. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  19. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 77
  20. 2018Discriminability objective for training descriptive captionscited by 67

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 3; 2025: 1.

10 papers.

  1. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Modelcited by 38
  2. 2023FoodLMM: A Versatile Food Assistant using Large Multi-modal Modelcited by 32
  3. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25
  4. 2023VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blendingcited by 17
  5. 2025Cluster-Aware Prompt Ensemble Learning for Few-Shot Vision-Language Model Adaptationcited by 7
  6. 2023SpliceMix: A Cross-scale and Semantic Blending Augmentation Strategy for Multi-label Image Classificationcited by 4
  7. 2023Keyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answeringcited by 3
  8. 2023Weakly-Supervised 3D Visual Grounding based on Visual Language Alignmentcited by 3
  9. 2024Lightweight Model Pre-training via Language Guided Knowledge Distillationcited by 1
  10. 2024InteractPro: A Unified Framework for Motion-Aware Image Compositioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024POINTS: Improving Your Vision-language Model with Affordable Strategies
  2. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
  3. 2026LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
  4. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
  5. 2022Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics
  6. 2024Unveiling Encoder-Free Vision-Language Models
  7. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  8. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision