mlsanityData to 14 Oct 2026

VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information

arXiv 2412.00947 ↗First posted 1 Dec 2024cs.CL cs.CVcited by 39

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large Vision Language Models (LVLMs) have achieved remarkable performance in various vision-language tasks. However, it is still unclear how accurately LVLMs can perceive visual information in images. In particular, the capability of LVLMs to perceive geometric information, such as shape, angle, and size, remains insufficiently analyzed, although the perception of these properties is crucial for tasks that require a detailed visual understanding. In this work, we introduce VisOnlyQA, a dataset for evaluating the geometric perception of LVLMs, and reveal that LVLMs often cannot accurately perceive basic geometric information in images, while human performance is nearly perfect. VisOnlyQA consists of 12 tasks that directly ask about geometric information in geometric shapes, charts, chemical structures, and 3D shapes. Our experiments highlight the following findings: (i) State-of-the-art LVLMs struggle with basic geometric perception. 23 LVLMs we evaluate, including GPT-4o and Gemini 2.5 Pro, work poorly on VisOnlyQA. (ii) Additional training data does not resolve this issue. Fine-tuning on the training set of VisOnlyQA is not always effective, even for in-distribution tasks. (iii) LLM may be the bottleneck. LVLMs using stronger LLMs exhibit better geometric perception on VisOnlyQA, while it does not require complex reasoning, suggesting that the way LVLMs process information from visual encoders is a bottleneck. The datasets, code, and model responses are provided at https://github.com/psunlpgroup/VisOnlyQA.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 79 references that are in the map. First posted on arXiv 1 Dec 2024.

Builds on

12 of 79 references.

References · 12

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  6. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  7. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  8. 2017ShapeWorld - A new test methodology for multimodal language understandingcited by 37
  9. 2023The Role of Chain-of-Thought in Complex Vision-Language Reasoning Taskcited by 32
  10. 2024Do Vision-Language Models Really Understand Visual Language?cited by 32
  11. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
  12. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 25; 2026: 13.

39 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  2. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 55
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  4. 2025Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understandingcited by 24
  5. 2025V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Modelscited by 22
  6. 2025VisNumBench: Evaluating Number Sense of Multimodal Large Language Modelscited by 14
  7. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 12
  8. 2026Phi-4-reasoning-vision-15B Technical Reportcited by 10
  9. 2025Towards Explainable Fake Image Detection with Multi-Modal Large Language Modelscited by 8
  10. 2025GeoPQA: Bridging the Visual Perception Gap in MLLMs for Geometric Reasoningcited by 8
  11. 2025VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMscited by 7
  12. 2025Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBenchcited by 7
27 more
  1. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 7
  2. 2025GeoDANO: Geometric VLM with Domain Agnostic Vision Encodercited by 6
  3. 2026From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Modelscited by 6
  4. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  5. 2025Decoupling the components of geometric understanding in Vision Language Modelscited by 4
  6. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  7. 2026OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Modelscited by 4
  8. 2025CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?cited by 3
  9. 2025Shape and Texture Recognition in Large Vision-Language Modelscited by 3
  10. 2025MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Modelscited by 3
  11. 2026Probing Visual Concepts in Lightweight Vision-Language Models for Automated Drivingcited by 3
  12. 2024Do large language vision models understand 3D shapes?cited by 2
  13. 2025Efficient Few-Shot Continual Learning in Vision-Language Modelscited by 2
  14. 2025Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutionscited by 2
  15. 2025Plane Geometry Problem Solving with Multi-modal Reasoning: A Surveycited by 2
  16. 2025Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perceptioncited by 2
  17. 2025CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updatescited by 1
  18. 2025The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?cited by 0
  19. 2026Why MLLMs Struggle to Determine Object Orientationscited by 0
  20. 2026State Beyond Appearance: Diagnosing and Improving State Consistency in Dial-Based Measurement Readingcited by 0
  21. 2026VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Followingcited by 0
  22. 2026Improving Selective Classification with Pairwise Queries for Binary Classificationcited by 0
  23. 2026A Dataset for Dynamic Human Preferences for Vision Language Modelscited by 0
  24. 2026Learning Where Outcomes Change:Credit-Addressable Reasoning for Multimodal Geometrycited by 0
  25. 2026Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structurescited by 0
  26. 2026Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentrationcited by 0
  27. 2026SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding Depth and Height Perception in Large Visual-Language Models
  2. 2024GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Models
  3. 2025CapGeo-Bench: Decoupling Visual Perception from Reasoning and Evaluating Geometric Understanding
  4. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions
  5. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation
  6. 2024Intriguing Properties of Large Language and Vision Models
  7. 2026GeoTikzBridge: Advancing Multimodal Code Generation for Geometric Perception and Reasoning
  8. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors