mlsanityData to 13 Oct 2026

Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench

arXiv 2510.26865 ↗First posted 30 Oct 2025cs.CV cs.AIcited by 6

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Reading measurement instruments is effortless for humans and requires relatively little domain expertise, yet it remains surprisingly challenging for current vision-language models (VLMs) as we find in preliminary evaluation. In this work, we introduce MeasureBench, a benchmark on visual measurement reading covering both real-world and synthesized images of various types of measurements, along with an extensible pipeline for data synthesis. Our pipeline procedurally generates a specified type of gauge with controllable visual appearance, enabling scalable variation in key details such as pointers, scales, fonts, lighting, and clutter. Evaluation on popular proprietary and open-weight VLMs shows that even the strongest frontier VLMs struggle with measurement reading in general. We have also conducted preliminary experiments with reinforcement finetuning (RFT) over synthetic data, and find a significant improvement on both in-domain synthetic subset and real-world images. Our analysis highlights a fundamental limitation of current VLMs in fine-grained spatial grounding. We hope this resource and our code releases can help future advances on visually grounded numeracy and precise spatial perception of VLMs, bridging the gap between recognizing numbers and measuring the world.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 43 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

31 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025CAD2DMD-SET: Synthetic Generation Tool of Digital Measurement Device CAD Model Datasets for fine-tuning Large Vision-Language Models29 Aug
  2. 2025Vision-Language Models Can't See the Obvious7 Jul
  3. 2024Vision language models are blind: Failing to translate detailed visual features into words9 Jul
  4. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
  5. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 25

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  2. 2023Visual Instruction Tuningcited by 7,827
  3. 2025Qwen2.5-VL Technical Reportcited by 5,499
  4. 2024GPT-4o System Cardcited by 4,536
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,714
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,546
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,118
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,896
  11. 2019Towards VQA Models That Can Readcited by 1,705
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,461
13 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,230
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 933
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 827
  4. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 502
  5. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  6. 2024VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Informationcited by 39
  7. 2021It's About Time: Analog Clock Reading in the Wildcited by 14
  8. 2025Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMscited by 10
  9. 2023Learning to Read Analog Gauges from Synthetic Datacited by 3
  10. 2025FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluationcited by 3
  11. 2022Image-based Automatic Dial Meter Reading in Unconstrained Scenarioscited by 2
  12. 2025RulerNet: Learning Perspective-Invariant Ruler Representations for Robust Image Scale Estimationcited by 1
  13. 2025FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questionscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenescited by 3
  2. 2026Lost in Motion: Vision Language Models Fail the Dynamic Gauges Testcited by 0
  3. 2026Handoff-H1: An Orchestrated Vision-Agent System for Material Quantity Takeoff from Construction Blueprintscited by 0
  4. 2026InSituMeasure: Probing Situated Measurement Grounding in Industrial Scenes with Multimodal Large Language Modelscited by 0
  5. 2026One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluationcited by 0
  6. 2026Type-Balanced Federated Learning for Visual Analog Meter Readingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring Primitive Visual Measurement Understanding and the Role of Output Format in Learning in Vision-Language Models
  2. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  3. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  4. 2025Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation
  5. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
  6. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  7. 2026Allegory of the Cave: Measurement-Grounded Vision-Language Learning
  8. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling