mlsanityData to 14 Oct 2026

Griffon: Spelling out All Object Locations at Any Granularity with Large Language Models

arXiv 2311.14552 ↗First posted 24 Nov 2023cs.CV cs.AIcited by 31

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Replicating the innate human ability to detect all objects based on free-form texts at any granularity remains a formidable challenge for Large Vision Language Models (LVLMs). Current LVLMs are predominantly constrained to locate a single, pre-existing object. This limitation leads to a compromise in model design, necessitating the introduction of visual expert models or customized head structures. Beyond these constraints, our research uncovers LVLMs' capability for basic object perception, allowing them to accurately identify and locate objects of interest. Building on this insight, we introduce a novel Language-prompted Localization Dataset to fully unleash the capabilities of LVLMs in fine-grained object perception and precise location awareness. More importantly, we present Griffon, a purely LVLM-based baseline, which does not introduce any special tokens, expert models, or additional detection modules. It simply maintains a consistent structure with popular LVLMs by unifying data formats across various localization-related scenarios and is trained end-to-end through a well-designed pipeline. Comprehensive experiments demonstrate that Griffon not only achieves state-of-the-art performance on the fine-grained RefCOCO series and Flickr30K Entities but also approaches the capabilities of the expert model Faster RCNN on the detection benchmark MSCOCO. Data, codes, and models are released at https://github.com/jefferyZhan/Griffon.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 72 references that are in the map. First posted on arXiv 24 Nov 2023.

Builds on

67 of 72 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  3. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  6. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Segment Anythingcited by 8,234
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2017Mask R-CNNcited by 7,277
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
38 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  7. 2015Fast R-CNNcited by 2,933
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2015VQA: Visual Question Answeringcited by 2,506
  13. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  14. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  15. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  16. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  17. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  18. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  19. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  20. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  21. 2021Grounded Language-Image Pre-trainingcited by 1,156
  22. 2016Modeling Context in Referring Expressionscited by 1,063
  23. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  24. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  25. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  26. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  27. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  28. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  29. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  30. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  31. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  32. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  33. 2016R-FCN: Object Detection via Region-based Fully Convolutional Networkscited by 414
  34. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  35. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 382
  36. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  37. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  38. 2018Bilinear Attention Networkscited by 280

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 14; 2026: 7.

31 papers.

Sort:

By area:

  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  2. 2025Detect Anything via Next Point Predictioncited by 73
  3. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 71
  4. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 69
  5. 2025LEGION: Learning to Ground and Explain for Synthetic Image Detectioncited by 65
  6. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  7. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 53
  8. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  9. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  10. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  11. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  12. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
19 more
  1. 2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Modelscited by 24
  2. 2025Referring to Any Personcited by 21
  3. 2025RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenescited by 19
  4. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  5. 2025WeDetect: Fast Open-Vocabulary Object Detection as Retrievalcited by 14
  6. 2025STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Groundingcited by 11
  7. 2025LMM-Det: Make Large Multimodal Models Excel in Object Detectioncited by 10
  8. 2024LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformercited by 8
  9. 2024Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Modelscited by 3
  10. 2025Spatial Preference Rewarding for MLLMs Spatial Understandingcited by 3
  11. 2025Grounding Everything in Tokens for Multimodal Large Language Modelscited by 2
  12. 2026Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Groundingcited by 1
  13. 2025MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answeringcited by 0
  14. 2025Logo-VGR: Visual Grounded Reasoning for Open-world Logo Recognitioncited by 0
  15. 2026Mastering Negation: Boosting Grounding Models via Grouped Opposition-Based Learningcited by 0
  16. 2026Consistency Beyond Contrast: Enhancing Open-Vocabulary Object Detection Robustness via Contextual Consistency Learningcited by 0
  17. 2026UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagerycited by 0
  18. 2026Assessing the Benefits of Combining Advanced Deep Learning Techniques for Post-Disaster Building Damage Assessment from UAV Imagerycited by 0
  19. 2026LOCI: A Locator-Critic with Refinement Loopcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mechanisms of Object Localization in Vision-Language Models
  2. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
  3. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
  4. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  5. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
  6. 2026SceneFunRI: Reasoning the Invisible for Task-Driven Functional Object Localization
  7. 2025Foundation Models for Remote Sensing: An Analysis of MLLMs for Object Localization
  8. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation