mlsanityData to 7 Oct 2026

Ferret: Refer and Ground Anything Anywhere at Any Granularity

arXiv 2310.07704 ↗First posted 11 Oct 2023cs.CV cs.CLcited by 498

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce Ferret, a new Multimodal Large Language Model (MLLM) capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions. To unify referring and grounding in the LLM paradigm, Ferret employs a novel and powerful hybrid region representation that integrates discrete coordinates and continuous features jointly to represent a region in the image. To extract the continuous features of versatile regions, we propose a spatial-aware visual sampler, adept at handling varying sparsity across different shapes. Consequently, Ferret can accept diverse region inputs, such as points, bounding boxes, and free-form shapes. To bolster the desired capability of Ferret, we curate GRIT, a comprehensive refer-and-ground instruction tuning dataset including 1.1M samples that contain rich hierarchical spatial knowledge, with 95K hard negative data to promote model robustness. The resulting model not only achieves superior performance in classical referring and grounding tasks, but also greatly outperforms existing MLLMs in region-based and localization-demanded multimodal chatting. Our evaluations also reveal a significantly improved capability of describing image details and a remarkable alleviation in object hallucination. Code and data will be available at https://github.com/apple/ml-ferret

Key facts. Cited by 498 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 75 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

66 of 75 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  3. 2023Contextual Object Detection with Multimodal Large Language Models29 May
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  5. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2017Mask R-CNNcited by 7,259
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  12. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  10. 2021Grounded Language-Image Pre-trainingcited by 1,147
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  13. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  14. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  15. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  16. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  17. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  18. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  19. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  20. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  21. 2023Segment Everything Everywhere All at Oncecited by 546
  22. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  23. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  24. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  25. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  26. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  27. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  28. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  29. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  30. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  31. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  32. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  33. 2023Generating Images with Multimodal Language Modelscited by 259
  34. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  35. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  36. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  37. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  38. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 191; 2025: 163; 2026: 110.

498 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  4. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  5. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  6. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  7. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  8. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  9. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  11. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  12. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
38 more
  1. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  3. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  4. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  5. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  6. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  7. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  8. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  9. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  10. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  11. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  12. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  13. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  14. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  15. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  16. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 136
  17. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  18. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  19. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  20. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  21. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  22. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  23. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  24. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  25. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  26. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
  27. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  28. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  29. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  30. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  31. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 96
  32. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  33. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  34. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  35. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  36. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 85
  37. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  38. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84

50 most cited of 498 All 498 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding
  2. 2026IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
  3. 2025Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement
  4. 2025C2F-Space: Coarse-to-Fine Space Grounding for Spatial Instructions using Vision-Language Models
  5. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model
  6. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  7. 2024ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension
  8. 20243D Spatial Understanding in MLLMs: Disambiguation and Evaluation