mlsanityData to 14 Oct 2026

GLIPv2: Unifying Localization and Vision-Language Understanding

arXiv 2206.05836 ↗First posted 12 Jun 2022cs.CV cs.AI cs.CLcited by 275

Not on the map. Why?

Cited by 57 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning). GLIPv2 elegantly unifies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as a VL reformulation of the detection task, region-word contrastive learning as a novel region-word level contrastive learning task, and the masked language modeling. This unification not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks. Experimental results show that a single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks. The model also shows (1) strong zero-shot and few-shot adaption performance on open-vocabulary object detection tasks and (2) superior grounding capability on VL understanding tasks. Code will be released at https://github.com/microsoft/GLIP.

Key facts. Cited by 275 papers in the map (data through 14 Oct 2026). Builds on 58 of its 78 references that are in the map. First posted on arXiv 12 Jun 2022.

Builds on

58 of 78 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2017Mask R-CNNcited by 7,277
  10. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2015VQA: Visual Question Answeringcited by 2,506
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  14. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  15. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  16. 2021Grounded Language-Image Pre-trainingcited by 1,156
  17. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  18. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  19. 2020On Layer Normalization in the Transformer Architecturecited by 922
  20. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 915
  21. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  22. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  23. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  24. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  25. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  26. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  27. 2019Hybrid Task Cascade for Instance Segmentationcited by 481
  28. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  29. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  30. 2016R-FCN: Object Detection via Region-based Fully Convolutional Networkscited by 414
  31. 2020Frustratingly Simple Few-Shot Object Detectioncited by 282
  32. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  33. 2019Zero-Shot Semantic Segmentationcited by 245
  34. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  35. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  36. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  37. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  38. 2018Zero-Shot Object Detectioncited by 189

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 21; 2023: 109; 2024: 92; 2025: 33; 2026: 20.

275 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  4. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  5. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  7. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  9. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  10. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  11. 2022Multimodal Learning with Transformers: A Surveycited by 400
  12. 2023Scaling Open-Vocabulary Object Detectioncited by 343
38 more
  1. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  2. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  3. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  4. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 274
  5. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  6. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  7. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  8. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 190
  9. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  10. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  11. 2023TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecastingcited by 174
  12. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  13. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 164
  14. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  15. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 145
  16. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  17. 2024Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detectioncited by 122
  18. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 121
  19. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 118
  20. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  21. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  22. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  23. 2024GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editingcited by 115
  24. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  25. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  26. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  27. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 109
  28. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  29. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  30. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  31. 2022Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignmentcited by 98
  32. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  33. 2024UniTS: A Unified Multi-Task Time Series Modelcited by 94
  34. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  35. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  36. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  37. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 89
  38. 2023CLIP the Gap: A Single Domain Generalization Approach for Object Detectioncited by 87

50 most cited of 275 All 275 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RegionGPT: Towards Region Understanding Vision Language Model
  2. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  3. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  5. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
  6. 2025Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection
  7. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation
  8. 2021VinVL: Revisiting Visual Representations in Vision-Language Models