# VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information

arXiv: https://arxiv.org/abs/2412.00947 · HTML: https://mlsanity.com/paper/2412.00947/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 12 of its 79 references that are in the map.
- First posted on arXiv 1 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023)
- [xGen-MM (BLIP-3): A Family of Open Large Multimodal Models](https://mlsanity.com/paper/2408.08872/) (2024)
- [ShapeWorld - A new test methodology for multimodal language understanding](https://mlsanity.com/paper/1704.04517/) (2017)
- [The Role of Chain-of-Thought in Complex Vision-Language Reasoning Task](https://mlsanity.com/paper/2311.09193/) (2023)
- [Do Vision-Language Models Really Understand Visual Language?](https://mlsanity.com/paper/2410.00193/) (2024)
- [Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark](https://mlsanity.com/paper/2410.14702/) (2024)
- [Eureka: Evaluating and Understanding Large Foundation Models](https://mlsanity.com/paper/2409.10566/) (2024)

## Built on by (most cited first, 30 of 39)

- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs](https://mlsanity.com/paper/2510.17771/) (2025)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding](https://mlsanity.com/paper/2502.11492/) (2025)
- [V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models](https://mlsanity.com/paper/2511.16668/) (2025)
- [VisNumBench: Evaluating Number Sense of Multimodal Large Language Models](https://mlsanity.com/paper/2503.14939/) (2025)
- [EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts](https://mlsanity.com/paper/2508.04650/) (2025)
- [Phi-4-reasoning-vision-15B Technical Report](https://mlsanity.com/paper/2603.03975/) (2026)
- [Towards Explainable Fake Image Detection with Multi-Modal Large Language Models](https://mlsanity.com/paper/2504.14245/) (2025)
- [GeoPQA: Bridging the Visual Perception Gap in MLLMs for Geometric Reasoning](https://mlsanity.com/paper/2509.17437/) (2025)
- [VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs](https://mlsanity.com/paper/2507.13361/) (2025)
- [Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench](https://mlsanity.com/paper/2510.26865/) (2025)
- [Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction](https://mlsanity.com/paper/2512.18813/) (2025)
- [GeoDANO: Geometric VLM with Domain Agnostic Vision Encoder](https://mlsanity.com/paper/2502.11360/) (2025)
- [From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models](https://mlsanity.com/paper/2605.20177/) (2026)
- [Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models](https://mlsanity.com/paper/2510.21978/) (2025)
- [Decoupling the components of geometric understanding in Vision Language Models](https://mlsanity.com/paper/2503.03840/) (2025)
- [A Survey of Deep Learning for Geometry Problem Solving](https://mlsanity.com/paper/2507.11936/) (2025)
- [OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2603.09326/) (2026)
- [CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?](https://mlsanity.com/paper/2502.11300/) (2025)
- [Shape and Texture Recognition in Large Vision-Language Models](https://mlsanity.com/paper/2503.23062/) (2025)
- [MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2508.06009/) (2025)
- [Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving](https://mlsanity.com/paper/2603.06054/) (2026)
- [Do large language vision models understand 3D shapes?](https://mlsanity.com/paper/2412.10908/) (2024)
- [Efficient Few-Shot Continual Learning in Vision-Language Models](https://mlsanity.com/paper/2502.04098/) (2025)
- [Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutions](https://mlsanity.com/paper/2502.04389/) (2025)
- [Plane Geometry Problem Solving with Multi-modal Reasoning: A Survey](https://mlsanity.com/paper/2505.14340/) (2025)
- [Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception](https://mlsanity.com/paper/2510.08352/) (2025)
- [CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates](https://mlsanity.com/paper/2512.10342/) (2025)
- [The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?](https://mlsanity.com/paper/2508.21143/) (2025)

## Abstract (arXiv)

Large Vision Language Models (LVLMs) have achieved remarkable performance in various vision-language tasks. However, it is still unclear how accurately LVLMs can perceive visual information in images. In particular, the capability of LVLMs to perceive geometric information, such as shape, angle, and size, remains insufficiently analyzed, although the perception of these properties is crucial for tasks that require a detailed visual understanding. In this work, we introduce VisOnlyQA, a dataset for evaluating the geometric perception of LVLMs, and reveal that LVLMs often cannot accurately perceive basic geometric information in images, while human performance is nearly perfect. VisOnlyQA consists of 12 tasks that directly ask about geometric information in geometric shapes, charts, chemical structures, and 3D shapes. Our experiments highlight the following findings: (i) State-of-the-art LVLMs struggle with basic geometric perception. 23 LVLMs we evaluate, including GPT-4o and Gemini 2.5 Pro, work poorly on VisOnlyQA. (ii) Additional training data does not resolve this issue. Fine-tuning on the training set of VisOnlyQA is not always effective, even for in-distribution tasks. (iii) LLM may be the bottleneck. LVLMs using stronger LLMs exhibit better geometric perception on VisOnlyQA, while it does not require complex reasoning, suggesting that the way LVLMs process information from visual encoders is a bottleneck. The datasets, code, and model responses are provided at https://github.com/psunlpgroup/VisOnlyQA.