# CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models arXiv: https://arxiv.org/abs/2505.17202 ยท HTML: https://mlsanity.com/paper/2505.17202/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 34 of its 80 references that are in the map. - First posted on arXiv 22 May 2025. ## Placed in - Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/) ## Builds on (most cited first, 30 of 34) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023) - [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022) - [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019) - [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023) - [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023) - [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022) - [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024) - [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022) - [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024) - [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019) - [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018) - [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024) - [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021) - [Vision language models are blind: Failing to translate detailed visual features into words](https://mlsanity.com/paper/2407.06581/) (2024) - [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022) - [ChartBench: A Benchmark for Complex Visual Reasoning in Charts](https://mlsanity.com/paper/2312.15915/) (2023) - [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022) - [From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models](https://mlsanity.com/paper/2403.12027/) (2024) - [ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering](https://mlsanity.com/paper/2405.07001/) (2024) - [Large language models predict human sensory judgments across six modalities](https://mlsanity.com/paper/2302.01308/) (2023) - [AGENT: A Benchmark for Core Psychological Reasoning](https://mlsanity.com/paper/2102.12321/) (2021) - [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021) - [Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning](https://mlsanity.com/paper/2407.20174/) (2024) - [SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction](https://mlsanity.com/paper/2312.03035/) (2023) - [Evaluating Multiview Object Consistency in Humans and Image Models](https://mlsanity.com/paper/2409.05862/) (2024) - [Visualization Literacy of Multimodal Large Language Models: A Comparative Study](https://mlsanity.com/paper/2407.10996/) (2024) - [Adaptive Assessment of Visualization Literacy](https://mlsanity.com/paper/2308.14147/) (2023) ## Built on by (most cited first, 5 of 5) - [GPT-5 Model Corrected GPT-4V's Chart Reading Errors, Not Prompting](https://mlsanity.com/paper/2510.06782/) (2025) - [Toward a Machine Bertin: Why Visualization Needs Design Principles for Machine Cognition](https://mlsanity.com/paper/2602.01527/) (2026) - [Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy](https://mlsanity.com/paper/2606.03142/) (2026) - [More Accurate, Less Human: Gestalt Grouping in Vision Models](https://mlsanity.com/paper/2608.10195/) (2026) - [LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventions](https://mlsanity.com/paper/2609.24172/) (2026) ## Abstract (arXiv) Data visualizations are powerful tools for communicating patterns in quantitative data. Yet understanding any data visualization is no small feat -- succeeding requires jointly making sense of visual, numerical, and linguistic inputs arranged in a conventionalized format one has previously learned to parse. Recently developed vision-language models are, in principle, promising candidates for developing computational models of these cognitive operations. However, it is currently unclear to what degree these models emulate human behavior on tasks that involve reasoning about data visualizations. This gap reflects limitations in prior work that has evaluated data visualization understanding in artificial systems using measures that differ from those typically used to assess these abilities in humans. Here we evaluated eight vision-language models on six data visualization literacy assessments designed for humans and compared model responses to those of human participants. We found that these models performed worse than human participants on average, and this performance gap persisted even when using relatively lenient criteria to assess model performance. Moreover, while relative performance across items was somewhat correlated between models and humans, all models produced patterns of errors that were reliably distinct from those produced by human participants. Taken together, these findings suggest significant opportunities for further development of artificial systems that might serve as useful models of how humans reason about data visualizations. All code and data needed to reproduce these results are available at: https://osf.io/e25mu/?view_only=399daff5a14d4b16b09473cf19043f18.