mlsanityData to 14 Oct 2026

StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning

arXiv 2606.00148 ↗First posted 29 May 2026cs.CV cs.AIcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) often know the rule but pick the wrong answer: on abstract visual reasoning (AVR) tasks, a model can describe what it sees and name the underlying pattern, yet still fail to choose the matching candidate. Existing AVR benchmarks cannot detect this because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal. We introduce StemBind, a shared-stem diagnostic benchmark that probes the same visual stem with three aligned questions: Perception (what is in the image), Rule (what pattern governs it), and Full (which option completes it), so a final-answer error can be attributed to a specific sub-step on the same evidence. StemBind contains 2,298 curated knowledge-light stems across nine auditable visual operations, totaling 19,533 P/R/F tasks, with each full item annotated by Sternberg's four reasoning stages (S1 Encode, S2 Infer, S3 Map, S4 Apply). Evaluating 24 frontier MLLM configurations yields four findings. (i) The R-F chasm: rule accuracy exceeds full-item accuracy on 22 of 24 models, so most failures happen after the rule is identified. (ii) A persistent binding gap: even when P and R are both correct on the same stem, models still answer F incorrectly 51.2% of the time. (iii) The bottleneck is S3: process diagnostics and Stage-wise Stimulus Augmentation localize the dominant failure to rule-to-instance mapping. (iv) Scaling and thinking do not help: neither larger models nor explicit thinking mode reliably closes the gap, and thinking even lowers rule and full-item accuracy. StemBind reframes AVR evaluation from final-answer ranking to locating where abstract visual reasoning breaks down, identifying rule-to-instance binding as a concrete next target for vision-grounded reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 56 of its 64 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

56 of 64 references.

Closest ancestors · 29

Same direction, newest first

  1. 2026OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model22 Apr
  2. 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs5 Feb
  3. 2026AdaptMMBench: Benchmarking Adaptive Multimodal Reasoning for Mode Selection and Reasoning Process2 Feb
  4. 2026CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solving5 Jan
  5. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs24 Dec
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency25 Aug
  7. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning1 Jul
  8. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers30 Jun
  9. 2025MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning5 Jun
  10. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark4 Jun
  11. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning3 Jun
  12. 2025Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task28 May
17 more
  1. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models21 May
  2. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought21 May
  3. 2025IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests17 May
  4. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models21 Apr
  5. 2025VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge14 Apr
  6. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoning13 Mar
  7. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency13 Feb
  8. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning18 Jun
  9. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models13 Jun
  10. 2024ReMI: A Dataset for Reasoning with Multiple Images13 Jun
  11. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought26 May
  12. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning21 Apr
  13. 2024MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classification7 Apr
  14. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns20 Mar
  15. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  16. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  17. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 27

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  8. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  9. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  10. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  11. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  12. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
15 more
  1. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  2. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  3. 2018Measuring abstract reasoning in neural networkscited by 166
  4. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 101
  5. 2020Stratified Rule-Aware Network for Abstract Visual Reasoningcited by 72
  6. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  7. 2020Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoningcited by 63
  8. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 39
  9. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  10. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Modelscited by 29
  11. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24
  12. 2024M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmarkcited by 14
  13. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 13
  14. 2025SpatialViz-Bench: A Cognitively-Grounded Benchmark for Diagnosing Spatial Visualization in MLLMscited by 12
  15. 2026ENC-Bench: A Benchmark for Evaluating Multimodal Large Language Models in Electronic Navigational Chart Understandingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks
  2. 2025CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs
  3. 2026VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
  4. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  5. 2025PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection
  6. 2025A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs
  7. 2026MM-OPD: Towards One More Bottleneck Between Perception and Reasoning
  8. 2025On Data Synthesis and Post-training for Visual Abstract Reasoning