mlsanityData to 7 Oct 2026

How Far Are We from Intelligent Visual Deductive Reasoning?

arXiv 2403.04732 ↗First posted 7 Mar 2024cs.AI cs.CL cs.CVcited by 29

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have recently demonstrated incredible strides on diverse vision language tasks. We dig into vision-based deductive reasoning, a more sophisticated but less explored realm, and find previously unexposed blindspots in the current SOTA VLMs. Specifically, we leverage Raven's Progressive Matrices (RPMs), to assess VLMs' abilities to perform multi-hop relational and deductive reasoning relying solely on visual clues. We perform comprehensive evaluations of several popular VLMs employing standard strategies such as in-context learning, self-consistency, and Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test, IntelligenceTest, and RAVEN. The results reveal that despite the impressive capabilities of LLMs in text-based reasoning, we are still far from achieving comparable proficiency in visual deductive reasoning. We found that certain standard strategies that are effective when applied to LLMs do not seamlessly translate to the challenges presented by visual reasoning tasks. A detailed analysis reveals that VLMs struggle to solve these tasks mainly because they are unable to perceive and comprehend multiple, confounding abstract patterns in RPM examples.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 37 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

19 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models22 Jan
  2. 2024Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imagination16 Jan
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  11. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  12. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
4 more
  1. 2022Holistic Evaluation of Language Modelscited by 115
  2. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  3. 2023LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyondcited by 37
  4. 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Gamescited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 13; 2026: 6.

29 papers.

Sort:

By area:

  1. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  2. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 69
  3. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Mergingcited by 40
  4. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  5. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMscited by 32
  6. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  7. 2024MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?cited by 28
  8. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  9. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?cited by 16
  10. 2025Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMscited by 14
  11. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  12. 2025Do AI Models Perform Human-like Abstract Reasoning Across Modalities?cited by 10
17 more
  1. 2024Image-to-Text Logic Jailbreak: Your Imagination can Help You Do Anythingcited by 7
  2. 2024Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoningcited by 7
  3. 2025Synthesizing Visual Concepts as Vision-Language Programscited by 6
  4. 2025Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?cited by 5
  5. 2025Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Taskcited by 5
  6. 2025I-RAVEN-X: Benchmarking Generalization and Robustness of Analogical and Mathematical Reasoning in Large Language and Reasoning Modelscited by 4
  7. 2026Toward Cognitive Supersensing in Multimodal Large Language Modelcited by 4
  8. 2024Chain-of-Sketch: Enabling Global Visual Reasoningcited by 3
  9. 2025Unraveling the geometry of visual relational reasoningcited by 3
  10. 2026Disentangling Perception and Reasoning in Multimodal LLMs via Reward Designcited by 2
  11. 2024Funny-Valen-Tine: Planning Solution Distribution Enhances Machine Abstract Reasoning Abilitycited by 1
  12. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  13. 2025DIO: Refining Mutual Information and Causal Chain to Enhance Machine Abstract Reasoning Abilitycited by 0
  14. 2026VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thoughtcited by 0
  15. 2026Can VLMs Reason Robustly? A Neuro-Symbolic Investigationcited by 0
  16. 2026Formalizing the Binding Problemcited by 0
  17. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Investigating Relational Reasoning in VLMs
  2. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  3. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  4. 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs
  5. 2025A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs
  6. 2026From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
  7. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models
  8. 2025VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs