mlsanityData to 14 Oct 2026

LocateBench: Evaluating the Locating Ability of Vision Language Models

arXiv 2410.19808 ↗First posted 17 Oct 2024cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The ability to locate an object in an image according to natural language instructions is crucial for many real-world applications. In this work we propose LocateBench, a high-quality benchmark dedicated to evaluating this ability. We experiment with multiple prompting approaches, and measure the accuracy of several large vision language models. We find that even the accuracy of the strongest model, GPT-4o, lags behind human accuracy by more than 10%.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 49 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

34 of 49 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
22 more
  1. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  2. 2016Modeling Context in Referring Expressionscited by 1,063
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  4. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  5. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  8. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  9. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  10. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  11. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  12. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  13. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  14. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  15. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  16. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  17. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  18. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  19. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 71
  20. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  21. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39
  22. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 38

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosiscited by 8
  2. 2025Comprehension of Multilingual Expressions Referring to Target Objects in Visual Inputscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
  2. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
  3. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Grounding
  4. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  5. 2024On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning
  6. 2025On the Domain Robustness of Contrastive Vision-Language Models
  7. 2025Can Large Vision Language Models Read Maps Like a Human?
  8. 2023Griffon: Spelling out All Object Locations at Any Granularity with Large Language Models