mlsanityData to 7 Oct 2026

Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models

arXiv 2406.16866 ↗First posted 24 Jun 2024cs.CVcited by 52

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Referring expression comprehension (REC) involves localizing a target instance based on a textual description. Recent advancements in REC have been driven by large multimodal models (LMMs) like CogVLM, which achieved 92.44% accuracy on RefCOCO. However, this study questions whether existing benchmarks such as RefCOCO, RefCOCO+, and RefCOCOg, capture LMMs' comprehensive capabilities. We begin with a manual examination of these benchmarks, revealing high labeling error rates: 14% in RefCOCO, 24% in RefCOCO+, and 5% in RefCOCOg, which undermines the authenticity of evaluations. We address this by excluding problematic instances and reevaluating several LMMs capable of handling the REC task, showing significant accuracy improvements, thus highlighting the impact of benchmark noise. In response, we introduce Ref-L4, a comprehensive REC benchmark, specifically designed to evaluate modern REC models. Ref-L4 is distinguished by four key features: 1) a substantial sample size with 45,341 annotations; 2) a diverse range of object categories with 365 distinct types and varying instance scales from 30 to 3,767; 3) lengthy referring expressions averaging 24.2 words; and 4) an extensive vocabulary comprising 22,813 unique words. We evaluate a total of 24 large models on Ref-L4 and provide valuable insights. The cleaned versions of RefCOCO, RefCOCO+, and RefCOCOg, as well as our Ref-L4 benchmark and evaluation code, are available at https://github.com/JierunChen/Ref-L4.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 83 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

41 of 83 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models19 Apr
  2. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model21 Mar
  3. 2024RelationVLM: Making Large Vision-Language Models Understand Visual Relations19 Mar
  4. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring14 Mar
  5. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Model11 Jan
  6. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model19 Dec
  7. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  8. 2023Lenna: Language Enhanced Reasoning Detection Assistant5 Dec
  9. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  10. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Models24 Nov
  11. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  12. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
3 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun

Other references · 26

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2024Mixtral of Expertscited by 1,785
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
14 more
  1. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  2. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  3. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  4. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  5. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  6. 2022Language Models are General-Purpose Interfacescited by 96
  7. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  8. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  9. 2024GiT: Towards Generalist Vision Transformer through Universal Language Interfacecited by 23
  10. 2024High-Quality Entity Segmentation and Groundingcited by 13
  11. 2020MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Groundingcited by 9
  12. 2024Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentionscited by 8
  13. 2024LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Groundingcited by 7
  14. 2022Towards Unifying Reference Expression Generation and Comprehensioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 27; 2026: 20.

52 papers.

Sort:

By area:

  1. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  2. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  3. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  4. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  5. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  6. 2024Toward Visual Grounding: A Surveycited by 52
  7. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  8. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  9. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  10. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  11. 2025Referring to Any Personcited by 20
  12. 2025FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answeringcited by 20
38 more
  1. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  2. 2025SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigationcited by 12
  3. 2025Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learningcited by 11
  4. 2025WeDetect: Fast Open-Vocabulary Object Detection as Retrievalcited by 11
  5. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 9
  6. 2024LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformercited by 8
  7. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
  8. 2026Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Taskscited by 8
  9. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluationcited by 7
  10. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  11. 2025CoT Referring: Improving Referring Expression Tasks with Grounded Reasoningcited by 5
  12. 2025The Mechanistic Emergence of Symbol Grounding in Language Modelscited by 5
  13. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 5
  14. 2026Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Modelscited by 5
  15. 2024CaLoRAify: Calorie Estimation with Visual-Text Pairing and LoRA-Driven Visual Language Modelscited by 4
  16. 2025UWBench: A Comprehensive Vision-Language Benchmark for Underwater Understandingcited by 4
  17. 2026Action-Sketcher: From Reasoning to Action via Visual Sketches for Long-Horizon Robotic Manipulationcited by 4
  18. 2026Steerable Visual Representationscited by 4
  19. 2025Multimodal Reference Visual Groundingcited by 3
  20. 2025JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Roboticscited by 3
  21. 2026Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Trainingcited by 2
  22. 2025Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Groundingcited by 1
  23. 2025The Role of Entropy in Visual Grounding: Analysis and Optimizationcited by 1
  24. 2026User Prompting Strategies and Prompt Enhancement Methods for Open-Set Object Detection in XR Environmentscited by 1
  25. 2026Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehensioncited by 1
  26. 2026Iterative Visual Thinking and the Self-Correction Mirage in VLM Groundingcited by 1
  27. 2025MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answeringcited by 0
  28. 2025CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understandingcited by 0
  29. 2025Zero-Shot Referring Expression Comprehension via Vison-Language True/False Verificationcited by 0
  30. 2025RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehensioncited by 0
  31. 2026ObjEmbed: Towards Universal Multimodal Object Embeddingscited by 0
  32. 2026Towards Motion-aware Referring Image Segmentationcited by 0
  33. 2026Heuristic-inspired Reasoning Priors Facilitate Data-Efficient Referring Object Detectioncited by 0
  34. 2026Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphscited by 0
  35. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMscited by 0
  36. 2026Listening makes Vision Clear for VLMscited by 0
  37. 2026ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generationcited by 0
  38. 2026An LMM for Precisely Grounding Elements in Documentscited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaboration
  2. 2025KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge
  3. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension
  4. 2026Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker
  5. 2025ColLab: A Collaborative Spatial Progressive Data Engine for Referring Expression Comprehension and Generation
  6. 2024LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension
  7. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
  8. 2024The Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge