mlsanityData to 7 Oct 2026

Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

arXiv 2111.08276 ↗First posted 16 Nov 2021cs.CL cs.CVcited by 255

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.

Key facts. Cited by 255 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 57 references that are in the map. First posted on arXiv 16 Nov 2021.

Builds on

47 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  3. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning3 Jun
  4. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
31 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2016Modeling Context in Referring Expressionscited by 1,054
  15. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  16. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  17. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  18. 2016Self-critical Sequence Training for Image Captioningcited by 544
  19. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  20. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  21. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  22. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  23. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  24. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  25. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  26. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  27. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  28. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  29. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  30. 2021A Survey on Green Deep Learningcited by 52
  31. 2021KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillationcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 36; 2023: 85; 2024: 73; 2025: 38; 2026: 23.

255 papers.

Sort:

By area:

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  7. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  8. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  9. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  10. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  11. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  12. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
38 more
  1. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  2. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  3. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  4. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  5. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 119
  6. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  7. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  8. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  9. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  10. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  11. 2023Detecting and Grounding Multi-Modal Media Manipulationcited by 98
  12. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  13. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  14. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  15. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  16. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  17. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  18. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Studycited by 66
  19. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  20. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  21. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  22. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  23. 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 55
  24. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  25. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  26. 2023PuMer: Pruning and Merging Tokens for Efficient Vision Language Modelscited by 53
  27. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  28. 2024Toward Visual Grounding: A Surveycited by 52
  29. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  30. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  31. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  32. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  33. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  34. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  35. 2023OvarNet: Towards Open-vocabulary Object Attribute Recognitioncited by 43
  36. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  37. 2025Visual Agentic AI for Spatial Reasoning with a Dynamic APIcited by 42
  38. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 40

50 most cited of 255 All 255 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  2. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  3. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models
  4. 2025XR-VLM: Cross-Relationship Modeling with Multi-part Prompts and Visual Features for Fine-Grained Recognition
  5. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models
  6. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretraining
  7. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  8. 2025Visual Representation Alignment for Multimodal Large Language Models