mlsanityData to 7 Oct 2026

PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns

arXiv 2403.13315 ↗First posted 20 Mar 2024cs.CVcited by 54

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large multimodal models extend the impressive capabilities of large language models by integrating multimodal understanding abilities. However, it is not clear how they can emulate the general intelligence and reasoning ability of humans. As recognizing patterns and abstracting concepts are key to general intelligence, we introduce PuzzleVQA, a collection of 2000 puzzle instances based on abstract patterns. With this dataset, we evaluate large multimodal models with abstract patterns based on fundamental concepts, including colors, numbers, sizes, and shapes. Through our experiments on state-of-the-art large multimodal models, we find that they are not able to generalize well to simple abstract patterns. Notably, GPT-4V achieves a score of 46.4% on single-concept puzzles, which shows that state-of-the-art models struggle on our dataset. To diagnose the reasoning challenges in large multimodal models, we progressively guide the models with our ground truth reasoning explanations for visual perception, inductive reasoning, and deductive reasoning. Our systematic analysis finds that the main bottlenecks of GPT-4V are weaker visual perception and inductive reasoning abilities. Through this work, we hope to shed light on the limitations of large multimodal models and how they can better emulate human cognitive processes in the future. Our data and code are available at https://puzzlevqa.github.io

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 31 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

19 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  12. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
6 more
  1. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  2. 2022Is GPT-3 a Good Data Annotator?cited by 208
  3. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 97
  4. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  5. 2019Human few-shot learning of compositional instructionscited by 47
  6. 2024A Vision Check-up for Language Modelscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 27; 2026: 20.

54 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 92
  4. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  5. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  6. 2025Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigmcited by 41
  7. 2024MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problemscited by 38
  8. 2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
  9. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  10. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 31
  11. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 31
  12. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 29
38 more
  1. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  2. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Taskscited by 19
  3. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Countingcited by 17
  4. 2025Retrieval-Based Interleaved Visual Chain-of-Thought in Real-World Driving Scenarioscited by 16
  5. 2025SAIL-VL2 Technical Reportcited by 14
  6. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 12
  7. 2025Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learnedcited by 12
  8. 2024ING-VP: MLLMs cannot Play Easy Vision-based Games Yetcited by 10
  9. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  10. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomaliescited by 8
  11. 2025PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehuntscited by 7
  12. 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thoughtcited by 7
  13. 2025Explaining Puzzle Solutions in Natural Language: An Exploratory Study on 6x6 Sudokucited by 5
  14. 2025LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Modelscited by 5
  15. 2026DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoningcited by 5
  16. 2025The Jumping Reasoning Curve? Tracking the Evolution of Reasoning Performance in GPT-[n] and o-[n] Models on Multimodal Puzzlescited by 4
  17. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Taskscited by 4
  18. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  19. 2026Toward Cognitive Supersensing in Multimodal Large Language Modelcited by 4
  20. 2025On Data Synthesis and Post-training for Visual Abstract Reasoningcited by 3
  21. 2025MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planningcited by 3
  22. 2026TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RLcited by 3
  23. 2026Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Docited by 3
  24. 2025SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuningcited by 2
  25. 2025Oedipus and the Sphinx: Benchmarking and Improving Visual Language Models for Complex Graphic Reasoningcited by 1
  26. 2026AEGIS: Exploring the Limit of World Knowledge Capabilities for Unified Mulitmodal Modelscited by 1
  27. 2026Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agentscited by 1
  28. 2026Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoningcited by 1
  29. 2025GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzlescited by 0
  30. 2025Beyond the Linear Separability Ceiling: Aligning Representations in VLMscited by 0
  31. 2025Seeing isn't Hearing: Benchmarking Vision Language Models at Interpreting Spectrogramscited by 0
  32. 2026GenPlanner: From Noise to Plans -- Emergent Reasoning in Flow Matching and Diffusion Modelscited by 0
  33. 2026Leum-VL Technical Reportcited by 0
  34. 2026StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoningcited by 0
  35. 2026Position: The Systemic Lack of Agency in Visual Reasoningcited by 0
  36. 2026Switch-Reasoner: Learn When to Think in Multitask Mixtures via Reinforcement Learningcited by 0
  37. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understandingcited by 0
  38. 2026DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipescited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles
  2. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning
  3. 2024Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning
  4. 2025PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection
  5. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning
  6. 2026MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods
  7. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelity
  8. 2025Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbation