mlsanityData to 7 Oct 2026

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

arXiv 2104.03135 ↗First posted 7 Apr 2021cs.CVcited by 187

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We study joint learning of Convolutional Neural Network (CNN) and Transformer for vision-language pre-training (VLPT) which aims to learn cross-modal alignments from millions of image-text pairs. State-of-the-art approaches extract salient image regions and align regions with words step-by-step. As region-based visual features usually represent parts of an image, it is challenging for existing vision-language models to fully understand the semantics from paired natural languages. In this paper, we propose SOHO to "See Out of tHe bOx" that takes a whole image as input, and learns vision-language representation in an end-to-end manner. SOHO does not require bounding box annotations which enables inference 10 times faster than region-based approaches. In particular, SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics. It is updated on-the-fly and utilized in our proposed pre-training task Masked Visual Modeling (MVM). We conduct experiments on four well-established vision-language tasks by following standard VLPT settings. In particular, SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text retrieval 5k test split, 1.5% accuracy on NLVR2 test-P split, 6.7% accuracy on SNLI-VE test split, respectively.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 50 references that are in the map. First posted on arXiv 7 Apr 2021.

Builds on

42 of 50 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2015VQA: Visual Question Answeringcited by 2,489
30 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2018Image Transformercited by 770
  11. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  12. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  13. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  14. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  15. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  17. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  18. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  19. 2018Bilinear Attention Networkscited by 277
  20. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  21. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  22. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  23. 2020Learning Visual Representations with Caption Annotationscited by 123
  24. 2017Multiple Instance Detection Network with Online Instance Classifier Refinementcited by 117
  25. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  26. 2019WSOD^2: Learning Bottom-up and Top-down Objectness Distillation for Weakly-supervised Object Detectioncited by 51
  27. 2018Visual Entailment Task for Visually-Grounded Language Learningcited by 36
  28. 2019Why are Adaptive Methods Good for Attention Models?cited by 35
  29. 2018Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captionscited by 22
  30. 2018Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Trainingcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 65; 2023: 61; 2024: 17; 2025: 14; 2026: 3.

187 papers.

Sort:

By area:

  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  3. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  5. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  6. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  7. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  8. 2022Multimodal Learning with Transformers: A Surveycited by 399
  9. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  10. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  11. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  12. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
38 more
  1. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  2. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  3. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  4. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
  5. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  6. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  7. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  8. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  9. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  10. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  11. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  12. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  13. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  14. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  15. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  16. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  17. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  18. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  19. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  20. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  21. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  22. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  23. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
  24. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  25. 2022Multi-modal Alignment using Representation Codebookcited by 63
  26. 2022Learning Granularity-Unified Representations for Text-to-Image Person Re-identificationcited by 62
  27. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  28. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  29. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  30. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  31. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  32. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  33. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  34. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  35. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  36. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 45
  37. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  38. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attentioncited by 44

50 most cited of 187 All 187 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
  2. 2026ConvCue: Complementary Visual Inductive Biases for Vision-Language Models
  3. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
  4. 2022VL-BEiT: Generative Vision-Language Pretraining
  5. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  6. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
  7. 2024Locality Alignment Improves Vision-Language Models
  8. 2023Effective End-to-End Vision Language Pretraining with Semantic Visual Loss