# Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task

arXiv: https://arxiv.org/abs/2505.21850 · HTML: https://mlsanity.com/paper/2505.21850/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 21 of 21)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2302.03084/) (2023)
- [NVLM: Open Frontier-Class Multimodal LLMs](https://mlsanity.com/paper/2409.11402/) (2024)
- [The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain](https://mlsanity.com/paper/2305.07141/) (2023)
- [Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning](https://mlsanity.com/paper/2010.00763/) (2020)
- [AI2D-RST: A multimodal corpus of 1000 primary school science diagrams](https://mlsanity.com/paper/1912.03879/) (2019)
- [MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning](https://mlsanity.com/paper/2404.13591/) (2024)
- [How Far Are We from Intelligent Visual Deductive Reasoning?](https://mlsanity.com/paper/2403.04732/) (2024)
- [CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense](https://mlsanity.com/paper/1908.02962/) (2019)
- [Improved Visual Grounding through Self-Consistent Explanations](https://mlsanity.com/paper/2312.04554/) (2023)
- [KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph](https://mlsanity.com/paper/2409.10921/) (2024)

## Built on by (most cited first, 5 of 5)

- [Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning](https://mlsanity.com/paper/2604.05756/) (2026)
- [Red Teaming Multimodal Language Models: Evaluating Harm Across Prompt Modalities and Models](https://mlsanity.com/paper/2509.15478/) (2025)
- [PROPA: Toward Process-level Optimization in Visual Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2511.10279/) (2025)
- [Alignment Drift in Multimodal LLMs: A Two-Phase, Longitudinal Evaluation of Harm Across Eight Model Releases](https://mlsanity.com/paper/2602.04739/) (2026)
- [StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning](https://mlsanity.com/paper/2606.00148/) (2026)

## Abstract (arXiv)

Current Multimodal Large Language Models (MLLMs) excel in general visual reasoning but remain underexplored in Abstract Visual Reasoning (AVR), which demands higher-order reasoning to identify abstract rules beyond simple perception. Existing AVR benchmarks focus on single-step reasoning, emphasizing the end result but neglecting the multi-stage nature of reasoning process. Past studies found MLLMs struggle with these benchmarks, but it doesn't explain how they fail. To address this gap, we introduce MultiStAR, a Multi-Stage AVR benchmark, based on RAVEN, designed to assess reasoning across varying levels of complexity. Additionally, existing metrics like accuracy only focus on the final outcomes while do not account for the correctness of intermediate steps. Therefore, we propose a novel metric, MSEval, which considers the correctness of intermediate steps in addition to the final outcomes. We conduct comprehensive experiments on MultiStAR using 17 representative close-source and open-source MLLMs. The results reveal that while existing MLLMs perform adequately on basic perception tasks, they continue to face challenges in more complex rule detection stages.