mlsanityData to 14 Oct 2026

Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models

arXiv 2404.13013 ↗First posted 19 Apr 2024cs.CV cs.AI cs.CLcited by 119

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded and fine-grained visual perception ability. Beyond holistic image understanding, Groma is adept at region-level tasks such as region captioning and visual grounding. Such capabilities are built upon a localized visual tokenization mechanism, where an image input is decomposed into regions of interest and subsequently encoded into region tokens. By integrating region tokens into user instructions and model responses, we seamlessly enable Groma to understand user-specified region inputs and ground its textual output to images. Besides, to enhance the grounded chat ability of Groma, we curate a visually grounded instruction dataset by leveraging the powerful GPT-4V and visual prompting techniques. Compared with MLLMs that rely on the language model or external module for localization, Groma consistently demonstrates superior performances in standard referring and grounding benchmarks, highlighting the advantages of embedding localization into image tokenization. Project page: https://groma-mllm.github.io/.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 64 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

57 of 64 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024RegionGPT: Towards Region Understanding Vision Language Model4 Mar
  2. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  3. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  6. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2023Segment Anythingcited by 8,234
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2017Mask R-CNNcited by 7,277
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
37 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  10. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  12. 2016Modeling Context in Referring Expressionscited by 1,063
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  14. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  15. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  16. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  17. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  18. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  19. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  20. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  21. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  22. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  23. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  24. 2023Segment Everything Everywhere All at Oncecited by 548
  25. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  26. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  27. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  28. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
  29. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  30. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  31. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 121
  32. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  33. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  34. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  35. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  36. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  37. 2023General Object Foundation Model for Images and Videos at Scalecited by 71

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 24; 2025: 69; 2026: 26.

119 papers.

Sort:

By area:

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  3. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  4. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  5. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  6. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  7. 2025Detect Anything via Next Point Predictioncited by 73
  8. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 70
  9. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  10. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 61
  11. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 59
  12. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 53
38 more
  1. 2024Toward Visual Grounding: A Surveycited by 52
  2. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 51
  3. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 40
  4. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  5. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  6. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  7. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 32
  8. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  9. 2024Large Language Model with Region-guided Referring and Grounding for CT Report Generationcited by 28
  10. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capabilitycited by 28
  11. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  12. 2024Next Patch Prediction for Autoregressive Visual Generationcited by 26
  13. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 26
  14. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  15. 2025Referring to Any Personcited by 21
  16. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  17. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 18
  18. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  19. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  20. 2024PartGLEE: A Foundation Model for Recognizing and Parsing Any Objectscited by 15
  21. 2025On the Robustness of GUI Grounding Models Against Image Attackscited by 14
  22. 2025WeDetect: Fast Open-Vocabulary Object Detection as Retrievalcited by 14
  23. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  24. 2025OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioningcited by 12
  25. 2025When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMscited by 12
  26. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentationcited by 11
  27. 2025RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuningcited by 11
  28. 2025Visual Intention Grounding for Egocentric Assistantscited by 11
  29. 2025DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenescited by 11
  30. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  31. 2025LMM-Det: Make Large Multimodal Models Excel in Object Detectioncited by 10
  32. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  33. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 9
  34. 2025CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videoscited by 9
  35. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 9
  36. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  37. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  38. 2025EagleVision: Object-level Attribute Multimodal LLM for Remote Sensingcited by 8

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  2. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring
  3. 2026IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
  4. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
  5. 2025Zoomer: Adaptive Image Focus Optimization for Black-box MLLM
  6. 2026GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models
  7. 2026PGT: Procedurally Generated Tasks for improving visual grounding in MLLMs
  8. 2026MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment