# Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench

arXiv: https://arxiv.org/abs/2510.26865 · HTML: https://mlsanity.com/paper/2510.26865/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 31)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024)
- [SEED-Bench-2: Benchmarking Multimodal Large Language Models](https://mlsanity.com/paper/2311.17092/) (2023)
- [VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information](https://mlsanity.com/paper/2412.00947/) (2024)
- [It's About Time: Analog Clock Reading in the Wild](https://mlsanity.com/paper/2111.09162/) (2021)
- [Vision-Language Models Can't See the Obvious](https://mlsanity.com/paper/2507.04741/) (2025)
- [Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMs](https://mlsanity.com/paper/2502.05092/) (2025)
- [Learning to Read Analog Gauges from Synthetic Data](https://mlsanity.com/paper/2308.14583/) (2023)
- [FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation](https://mlsanity.com/paper/2506.09081/) (2025)
- [Image-based Automatic Dial Meter Reading in Unconstrained Scenarios](https://mlsanity.com/paper/2201.02850/) (2022)
- [RulerNet: Learning Perspective-Invariant Ruler Representations for Robust Image Scale Estimation](https://mlsanity.com/paper/2507.07077/) (2025)
- [CAD2DMD-SET: Synthetic Generation Tool of Digital Measurement Device CAD Model Datasets for fine-tuning Large Vision-Language Models](https://mlsanity.com/paper/2508.21732/) (2025)

## Built on by (most cited first, 7 of 7)

- [Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes](https://mlsanity.com/paper/2602.00593/) (2026)
- [Lost in Motion: Vision Language Models Fail the Dynamic Gauges Test](https://mlsanity.com/paper/2604.22829/) (2026)
- [Handoff-H1: An Orchestrated Vision-Agent System for Material Quantity Takeoff from Construction Blueprints](https://mlsanity.com/paper/2608.15032/) (2026)
- [InSituMeasure: Probing Situated Measurement Grounding in Industrial Scenes with Multimodal Large Language Models](https://mlsanity.com/paper/2609.04014/) (2026)
- [One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluation](https://mlsanity.com/paper/2609.06190/) (2026)
- [Type-Balanced Federated Learning for Visual Analog Meter Reading](https://mlsanity.com/paper/2609.31998/) (2026)
- [RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work](https://mlsanity.com/paper/2610.14397/) (2026)

## Abstract (arXiv)

Reading measurement instruments is effortless for humans and requires relatively little domain expertise, yet it remains surprisingly challenging for current vision-language models (VLMs) as we find in preliminary evaluation. In this work, we introduce MeasureBench, a benchmark on visual measurement reading covering both real-world and synthesized images of various types of measurements, along with an extensible pipeline for data synthesis. Our pipeline procedurally generates a specified type of gauge with controllable visual appearance, enabling scalable variation in key details such as pointers, scales, fonts, lighting, and clutter. Evaluation on popular proprietary and open-weight VLMs shows that even the strongest frontier VLMs struggle with measurement reading in general. We have also conducted preliminary experiments with reinforcement finetuning (RFT) over synthetic data, and find a significant improvement on both in-domain synthetic subset and real-world images. Our analysis highlights a fundamental limitation of current VLMs in fine-grained spatial grounding. We hope this resource and our code releases can help future advances on visually grounded numeracy and precise spatial perception of VLMs, bridging the gap between recognizing numbers and measuring the world.