mlsanityData to 7 Oct 2026

Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models

arXiv 2404.07973 ↗First posted 11 Apr 2024cs.CVcited by 97

Not on the map. Why?

Cited by 53 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks. In this work, we unveil Ferret-v2, a significant upgrade to Ferret, with three key designs. (1) Any resolution grounding and referring: A flexible approach that effortlessly handles higher image resolution, improving the model's ability to process and understand images in greater detail. (2) Multi-granularity visual encoding: By integrating the additional DINOv2 encoder, the model learns better and diverse underlying contexts for global and fine-grained visual information. (3) A three-stage training paradigm: Besides image-caption alignment, an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Experiments show that Ferret-v2 provides substantial improvements over Ferret and other state-of-the-art methods, thanks to its high-resolution scaling and fine-grained visual processing.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 67 of its 71 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

67 of 71 references.

References · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2016Modeling Context in Referring Expressionscited by 1,054
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  14. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  16. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  17. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  18. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  19. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  20. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  21. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  22. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  23. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  24. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  25. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  26. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  27. 2023Segment Everything Everywhere All at Oncecited by 546
  28. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  29. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  30. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  31. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  32. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  33. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  34. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  35. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  36. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  37. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  38. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 42; 2026: 23.

97 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  6. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  7. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  8. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  9. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  10. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  11. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  12. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
38 more
  1. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  2. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  3. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  4. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  5. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  6. 2025Detect Anything via Next Point Predictioncited by 66
  7. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  8. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  9. 2024Toward Visual Grounding: A Surveycited by 52
  10. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  11. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  12. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  13. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 39
  14. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  15. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  16. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  17. 2024Contrastive Localized Language-Image Pre-Trainingcited by 33
  18. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  19. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  20. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  21. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  22. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 26
  23. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  24. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 24
  25. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 23
  26. 2024WeatherQA: Can Multimodal Language Models Reason about Severe Weather?cited by 21
  27. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practicescited by 21
  28. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  29. 2025Referring to Any Personcited by 20
  30. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  31. 2024Law of Vision Representation in MLLMscited by 16
  32. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 16
  33. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 14
  34. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 14
  35. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  36. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  37. 2024TinyClick: Single-Turn Agent for Empowering GUI Automationcited by 12
  38. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMscited by 12

50 most cited of 97 All 97 (JSON)