# MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning

arXiv: https://arxiv.org/abs/2404.13591 · HTML: https://mlsanity.com/paper/2404.13591/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 47 of its 85 references that are in the map.
- First posted on arXiv 21 Apr 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Visual Entailment: A Novel Task for Fine-Grained Image Understanding](https://mlsanity.com/paper/1901.06706/) (2019)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model](https://mlsanity.com/paper/2312.11370/) (2023)
- [MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning](https://mlsanity.com/paper/2309.07915/) (2023)

## Built on by (most cited first, 30 of 30)

- [Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention](https://mlsanity.com/paper/2510.02912/) (2025)
- [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024)
- [Core Knowledge Deficits in Multi-Modal Language Models](https://mlsanity.com/paper/2410.10855/) (2024)
- [What is the Visual Cognition Gap between Humans and Multimodal LLMs?](https://mlsanity.com/paper/2406.10424/) (2024)
- [MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models](https://mlsanity.com/paper/2502.00698/) (2025)
- [HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks](https://mlsanity.com/paper/2410.12381/) (2024)
- [MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering](https://mlsanity.com/paper/2503.18491/) (2025)
- [MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models](https://mlsanity.com/paper/2410.09542/) (2024)
- [Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions](https://mlsanity.com/paper/2412.08737/) (2024)
- [NeSyGeo: A Neuro-Symbolic Framework for Multimodal Geometric Reasoning Data Generation](https://mlsanity.com/paper/2505.17121/) (2025)
- [NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models](https://mlsanity.com/paper/2407.10380/) (2024)
- [LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models](https://mlsanity.com/paper/2505.15616/) (2025)
- [Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning](https://mlsanity.com/paper/2412.05586/) (2024)
- [COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes](https://mlsanity.com/paper/2409.04053/) (2024)
- [Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?](https://mlsanity.com/paper/2503.11207/) (2025)
- [Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task](https://mlsanity.com/paper/2505.21850/) (2025)
- [The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs](https://mlsanity.com/paper/2410.01417/) (2024)
- [VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning](https://mlsanity.com/paper/2503.00043/) (2025)
- [I-RAVEN-X: Benchmarking Generalization and Robustness of Analogical and Mathematical Reasoning in Large Language and Reasoning Models](https://mlsanity.com/paper/2510.17496/) (2025)
- [On Data Synthesis and Post-training for Visual Abstract Reasoning](https://mlsanity.com/paper/2504.01324/) (2025)
- [VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs](https://mlsanity.com/paper/2512.21194/) (2025)
- [HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decoding](https://mlsanity.com/paper/2601.08273/) (2026)
- [SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models](https://mlsanity.com/paper/2609.01004/) (2026)
- [RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking](https://mlsanity.com/paper/2409.17458/) (2024)
- [Aligning Generalisation Between Humans and Machines](https://mlsanity.com/paper/2411.15626/) (2024)
- [Visual Reasoning Benchmark: Evaluating Multimodal LLMs on Classroom-Authentic Visual Problems from Primary Education](https://mlsanity.com/paper/2602.12196/) (2026)
- [MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning](https://mlsanity.com/paper/2603.12266/) (2026)
- [PROPA: Toward Process-level Optimization in Visual Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2511.10279/) (2025)
- [TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs](https://mlsanity.com/paper/2507.21584/) (2025)
- [StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning](https://mlsanity.com/paper/2606.00148/) (2026)

## Abstract (arXiv)

While multi-modal large language models (MLLMs) have shown significant progress on many popular visual reasoning benchmarks, whether they possess abstract visual reasoning abilities remains an open question. Similar to the Sudoku puzzles, abstract visual reasoning (AVR) problems require finding high-level patterns (e.g., repetition constraints) that control the input shapes (e.g., digits) in a specific task configuration (e.g., matrix). However, existing AVR benchmarks only considered a limited set of patterns (addition, conjunction), input shapes (rectangle, square), and task configurations (3 by 3 matrices). To evaluate MLLMs' reasoning abilities comprehensively, we introduce MARVEL, a multidimensional AVR benchmark with 770 puzzles composed of six core knowledge patterns, geometric and abstract shapes, and five different task configurations. To inspect whether the model accuracy is grounded in perception and reasoning, MARVEL complements the general AVR question with perception questions in a hierarchical evaluation framework. We conduct comprehensive experiments on MARVEL with nine representative MLLMs in zero-shot and few-shot settings. Our experiments reveal that all models show near-random performance on the AVR question, with significant performance gaps (40%) compared to humans across all patterns and task configurations. Further analysis of perception questions reveals that MLLMs struggle to comprehend the visual features (near-random performance) and even count the panels in the puzzle ( <45%), hindering their ability for abstract reasoning. We release our entire code and dataset.