mlsanityData to 14 Oct 2026

Vision language models are blind: Failing to translate detailed visual features into words

arXiv 2407.06581 ↗First posted 9 Jul 2024cs.AI cs.CVcited by 147

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

While large language models with vision capabilities (VLMs), e.g., GPT-4o and Gemini 1.5 Pro, score high on many vision-understanding benchmarks, they are still struggling with low-level vision tasks that are easy to humans. Specifically, on BlindTest, our suite of 7 very simple tasks, including identifying (a) whether two circles overlap; (b) how many times two lines intersect; (c) which letter is being circled in a word; and (d) the number of circles in an Olympic-like logo, four state-of-the-art VLMs are only 58.07% accurate on average. Claude 3.5 Sonnet performs the best at 77.84% accuracy, far from the human expected accuracy of 100%. Across different image resolutions and line widths, VLMs including slow-thinking models consistently struggle with those tasks that require precise spatial information when geometric primitives overlap or are close. Yet, VLMs perform at near-100% accuracy when much more space is added to separate shapes and letters. Linear probing experiments show that vision encoders contain sufficient visual information to solve BlindTest and that language models fail to decode this information into correct answers. Code and data are at: https://vlmsareblind.github.io

Key facts. Cited by 147 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 49 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

32 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context8 May
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  3. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  4. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 27

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
15 more
  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  2. 2016A Diagram Is Worth A Dozen Imagescited by 804
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  5. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  6. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  8. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  9. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  10. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  11. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 55
  12. 2024SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmarkcited by 35
  13. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMscited by 29
  14. 2023GlitchBench: Can large multimodal models detect video game glitches?cited by 23
  15. 2023Analyzing the Efficacy of an LLM-Only Approach for Image-based Document Question Answeringcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 70; 2026: 56.

147 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  2. 2025Do generative video models understand physical principles?cited by 161
  3. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  4. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 79
  5. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 75
  6. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 71
  7. 2026MIRAGE: The Illusion of Visual Understandingcited by 65
  8. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 58
  9. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  10. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 47
  11. 2025Kwai Keye-VL Technical Reportcited by 47
  12. 2025DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoningcited by 46
38 more
  1. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  2. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answeringcited by 37
  3. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
  4. 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
  5. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  6. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  7. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  8. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 24
  9. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24
  10. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  11. 2025Mull-Tokens: Modality-Agnostic Latent Thinkingcited by 19
  12. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 19
  13. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 18
  14. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?cited by 16
  15. 2025LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perceptioncited by 16
  16. 2025Is Your Video Language Model a Reliable Judge?cited by 14
  17. 2025How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Taskscited by 14
  18. 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 14
  19. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 14
  20. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  21. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 13
  22. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 13
  23. 2025Visual symbolic mechanisms: Emergent symbol processing in vision language modelscited by 13
  24. 2025Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Imagescited by 13
  25. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 12
  26. 2025XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoningcited by 12
  27. 2024Vision Language Models See What You Want but not What You Seecited by 10
  28. 2024Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Modelscited by 10
  29. 2025FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulationcited by 10
  30. 2025MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problemscited by 10
  31. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 10
  32. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
  33. 2026Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Groundingcited by 10
  34. 2025Unified Multimodal Understanding via Byte-Pair Visual Encodingcited by 9
  35. 2025Self-alignment of Large Video Language Models with Refined Regularized Preference Optimizationcited by 7
  36. 2025Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBenchcited by 7
  37. 2025Direct Visual Grounding by Directing Attention of Visual Tokenscited by 7
  38. 2026SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMscited by 7

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities
  2. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities
  3. 2024VLM's Eye Examination: Instruct and Inspect Visual Competency of Vision Language Models
  4. 2024Are VLMs Really Blind
  5. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting
  6. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  7. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  8. 2026Exploring the Use of VLMs for Navigation Assistance for People with Blindness and Low Vision