mlsanityData to 7 Oct 2026

VinVL: Revisiting Visual Representations in Vision-Language Models

arXiv 2101.00529 ↗First posted 2 Jan 2021cs.CV cs.AI cs.CLcited by 109

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric representations of images. Compared to the most widely used bottom-up and top-down model citeanderson2018bottom, the new model is bigger, better-designed for VL tasks, and pre-trained on much larger training corpora that combine multiple public annotated object detection datasets. Therefore, it can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses mainly on improving the vision-language fusion model and leaves the object detection model improvement untouched, we show that visual features matter significantly in VL models. In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model oscar citeli2020oscar, and utilize an improved approach short to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks. Our results show that the new visual features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. We will release the new object detection model to public.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 47 references that are in the map. First posted on arXiv 2 Jan 2021.

Builds on

36 of 47 references.

References · 36

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2016Self-critical Sequence Training for Image Captioningcited by 544
24 more
  1. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  2. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  3. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  4. 2018nocaps: novel object captioning at scalecited by 444
  5. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  6. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  7. 2019Attention on Attention for Image Captioningcited by 289
  8. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  9. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  10. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  11. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  12. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  13. 2020X-Linear Attention Networks for Image Captioningcited by 198
  14. 2020FCOS: A simple and strong anchor-free object detectorcited by 171
  15. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  16. 2017Dual-Path Convolutional Image-Text Embeddings with Instance Losscited by 145
  17. 2019Learning by Abstraction: The Neural State Machinecited by 113
  18. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  19. 2014From Captions to Visual Concepts and Backcited by 109
  20. 2020InterBERT: Vision-and-Language Interaction for Multi-modal Pretrainingcited by 48
  21. 2019Position Focused Attention Network for Image-Text Matchingcited by 45
  22. 2019Meta Module Network for Compositional Visual Reasoningcited by 39
  23. 2020VIVO: Visual Vocabulary Pre-Training for Novel Object Captioningcited by 30
  24. 2017VSE++: Improving Visual-Semantic Embeddings with Hard Negativescited by 20

Built on by

Cited by year

Citing papers per year: 2021: 33; 2022: 30; 2023: 35; 2024: 8; 2025: 3.

109 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  6. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  7. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  8. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  9. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  10. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  11. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  12. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
38 more
  1. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  3. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  4. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  5. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  6. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 90
  7. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  8. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84
  9. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  10. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  11. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  12. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  13. 2021Human-Adversarial Visual Question Answeringcited by 57
  14. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  15. 2021Towards General Purpose Vision Systemscited by 46
  16. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  17. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 45
  18. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  19. 2021Language Grounding with 3D Objectscited by 43
  20. 2021Transparent Human Evaluation for Image Captioningcited by 40
  21. 2023Improved Probabilistic Image-Text Representationscited by 40
  22. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  23. 2022ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCOcited by 37
  24. 2025FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmarkcited by 37
  25. 2021Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Modelcited by 34
  26. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
  27. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  28. 2022On Guiding Visual Attention with Language Specificationcited by 30
  29. 2023Retrieval-augmented Image Captioningcited by 28
  30. 2025IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layoutcited by 28
  31. 2022Position-guided Text Prompt for Vision-Language Pre-trainingcited by 27
  32. 2023DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulationcited by 27
  33. 2023EDIS: Entity-Driven Image Search over Multimodal Web Contentcited by 27
  34. 2022GRIT: General Robust Image Task Benchmarkcited by 26
  35. 2023Too Large; Data Reduction for Vision-Language Pre-Trainingcited by 26
  36. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  37. 2023TextFusion: Unveiling the Power of Textual Semantics for Controllable Image Fusioncited by 26
  38. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation
  2. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
  3. 2023Vision-Language Models for Vision Tasks: A Survey
  4. 2025Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models
  5. 2025A Review of 3D Object Detection with Vision-Language Models
  6. 2024Rethinking VLMs and LLMs for Image Classification
  7. 2024TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection
  8. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding