mlsanityData to 7 Oct 2026

GLaMM: Pixel Grounding Large Multimodal Model

arXiv 2311.03356 ↗First posted 6 Nov 2023cs.CV cs.AIcited by 447

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations.

Key facts. Cited by 447 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 65 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

56 of 65 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  4. 2023Tag2Text: Guiding Vision-Language Model via Image Tagging10 Mar
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  6. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2017Mask R-CNNcited by 7,259
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
37 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  6. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  13. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  14. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  15. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  17. 2018nocaps: novel object captioning at scalecited by 444
  18. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  19. 2023Segment Anything in High Qualitycited by 373
  20. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  21. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  22. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  23. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  24. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  25. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 273
  26. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  27. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  28. 2022Panoptic Scene Graph Generationcited by 114
  29. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  30. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  31. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  32. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  33. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  34. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  35. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  36. 2023Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognitioncited by 35
  37. 2022BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-trainingcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 118; 2025: 196; 2026: 125.

447 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  5. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  6. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  7. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  8. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  9. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  10. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  11. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  12. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
38 more
  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  2. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  3. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  4. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  5. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  6. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  7. 2024FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Modelscited by 111
  8. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  9. 2025GR-3 Technical Reportcited by 110
  10. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
  11. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  12. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  13. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  14. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  15. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  16. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 85
  17. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  18. 2024Benchmarking and Improving Detail Image Captioncited by 81
  19. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  20. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  21. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 72
  22. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  23. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  24. 2024TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Datacited by 69
  25. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  26. 2025LEGION: Learning to Ground and Explain for Synthetic Image Detectioncited by 65
  27. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
  28. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  29. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  30. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  31. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  32. 2024Language Repository for Long Video Understandingcited by 55
  33. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  34. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  35. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  36. 2024Toward Visual Grounding: A Surveycited by 52
  37. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  38. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50

50 most cited of 447 All 447 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024F-LMM: Grounding Frozen Large Multimodal Models
  2. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
  3. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
  4. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models
  5. 2026GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models
  6. 2025GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
  7. 2025Grounded Chain-of-Thought for Multimodal Large Language Models
  8. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation