# See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs

arXiv: https://arxiv.org/abs/2504.09809 · HTML: https://mlsanity.com/paper/2504.09809/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 32)

- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision](https://mlsanity.com/paper/2309.14181/) (2023)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions](https://mlsanity.com/paper/2308.09936/) (2023)
- [ChartBench: A Benchmark for Complex Visual Reasoning in Charts](https://mlsanity.com/paper/2312.15915/) (2023)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering](https://mlsanity.com/paper/2405.07001/) (2024)
- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [Visualization Generation with Large Language Models: An Evaluation](https://mlsanity.com/paper/2401.11255/) (2024)
- [How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?](https://mlsanity.com/paper/2407.17291/) (2024)
- [Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning](https://mlsanity.com/paper/2407.20174/) (2024)
- [Chart Question Answering: State of the Art and Future Directions](https://mlsanity.com/paper/2205.03966/) (2022)
- [Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis](https://mlsanity.com/paper/2412.03665/) (2024)
- [AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Making](https://mlsanity.com/paper/2312.04494/) (2023)
- [Exploring the Capability of LLMs in Performing Low-Level Visual Analytic Tasks on SVG Data Visualizations](https://mlsanity.com/paper/2404.19097/) (2024)
- [Mini-VLAT: A Short and Effective Measure of Visualization Literacy](https://mlsanity.com/paper/2304.07905/) (2023)
- [Beyond Generating Code: Evaluating GPT on a Data Visualization Course](https://mlsanity.com/paper/2306.02914/) (2023)
- [Are LLMs ready for Visualization?](https://mlsanity.com/paper/2403.06158/) (2024)
- [Rich Screen Reader Experiences for Accessible Data Visualization](https://mlsanity.com/paper/2205.04917/) (2022)
- [DracoGPT: Extracting Visualization Design Preferences from Large Language Models](https://mlsanity.com/paper/2408.06845/) (2024)
- [Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization](https://mlsanity.com/paper/2312.10610/) (2023)
- [How Aligned are Human Chart Takeaways and LLM Predictions? A Case Study on Bar Charts with Varying Layouts](https://mlsanity.com/paper/2408.06837/) (2024)
- [Can GPT-4 Models Detect Misleading Visualizations?](https://mlsanity.com/paper/2408.12617/) (2024)
- [Visualization Literacy of Multimodal Large Language Models: A Comparative Study](https://mlsanity.com/paper/2407.10996/) (2024)
- [Toward automatic comparison of visualization techniques: Application to graph visualization](https://mlsanity.com/paper/1910.09477/) (2019)

## Built on by (most cited first, 7 of 7)

- [Self-Rewarding Vision-Language Model via Reasoning Decomposition](https://mlsanity.com/paper/2508.19652/) (2025)
- [GeoVLM-R1: Reinforcement Fine-Tuning for Improved Remote Sensing Reasoning](https://mlsanity.com/paper/2509.25026/) (2025)
- [ParaView-MCP: An Autonomous Visualization Agent with Direct Tool Use](https://mlsanity.com/paper/2505.07064/) (2025)
- [Toward AI VIS Co-Scientists: A General and End-to-End Agent Harness for Solving Complex Data Visualization Tasks](https://mlsanity.com/paper/2605.21825/) (2026)
- [Do MLLMs See What We See? Analyzing Visualization Literacy Barriers in AI Systems](https://mlsanity.com/paper/2601.12585/) (2026)
- [Visual Attention Reasoning via Hierarchical Search and Self-Verification](https://mlsanity.com/paper/2510.18619/) (2025)
- [IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning](https://mlsanity.com/paper/2605.23997/) (2026)

## Abstract (arXiv)

Recent developments in multimodal large language models (MLLM) have equipped language models to reason about vision and language jointly. This permits MLLMs to both perceive and answer questions about data visualization across a variety of designs and tasks. Applying MLLMs to a broad range of visualization tasks requires us to properly evaluate their capabilities, and the most common way to conduct evaluation is through measuring a model's visualization reasoning capability, analogous to how we would evaluate human understanding of visualizations (e.g., visualization literacy). However, we found that in the context of visualization question answering (VisQA), how an MLLM perceives and reasons about visualizations can be fundamentally different from how humans approach the same problem. During the evaluation, even without visualization, the model could correctly answer a substantial portion of the visualization test questions, regardless of whether any selection options were provided. We hypothesize that the vast amount of knowledge encoded in the language model permits factual recall that supersedes the need to seek information from the visual signal. It raises concerns that the current VisQA evaluation may not fully capture the models' visualization reasoning capabilities. To address this, we propose a comprehensive sanity check framework that integrates a rule-based decision tree and a sanity check table to disentangle the effects of "seeing" (visual processing) and "recall" (reliance on prior knowledge). This validates VisQA datasets for evaluation, highlighting where models are truly "seeing", positively or negatively affected by the factual recall, or relying on inductive biases for question answering. Our study underscores the need for careful consideration in designing future visualization understanding studies when utilizing MLLMs.