Beyond Visual Understanding: Introducing PARROT-360V for Vision Language Model Benchmarking
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Current benchmarks for evaluating Vision Language Models (VLMs) often fall short in thoroughly assessing model abilities to understand and process complex visual and textual content. They typically focus on simple tasks that do not require deep reasoning or the integration of multiple data modalities to solve an original problem. To address this gap, we introduce the PARROT-360V Benchmark, a novel and comprehensive benchmark featuring 2487 challenging visual puzzles designed to test VLMs on complex visual reasoning tasks. We evaluated leading models: GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro, using PARROT-360V to assess their capabilities in combining visual clues with language skills to solve tasks in a manner akin to human problem-solving. Our findings reveal a notable performance gap: state-of-the-art models scored between 28 to 56 percentage on our benchmark, significantly lower than their performance on popular benchmarks. This underscores the limitations of current VLMs in handling complex, multi-step reasoning tasks and highlights the need for more robust evaluation frameworks to advance the field.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 16 references that are in the map. First posted on arXiv 20 Nov 2024.
Builds on
7 of 16 references.
Closest ancestors · 1
Same direction, newest first
Other references · 6
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2016A Diagram Is Worth A Dozen Imagescited by 793
- 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
- 2024How Susceptible are LLMs to Influence in Prompts?cited by 32
- 2024Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challengescited by 18
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs
- 2025VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Models
- 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
- 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
- 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models
- 2024VHELM: A Holistic Evaluation of Vision Language Models
- 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanese
- 2025Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark