mlsanityData to 7 Oct 2026

Benchmarking Visual Language Models on Standardized Visualization Literacy Tests

arXiv 2503.16632 ↗First posted 20 Mar 2025cs.HCcited by 20

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

The increasing integration of Visual Language Models (VLMs) into visualization systems demands a comprehensive understanding of their visual interpretation capabilities and constraints. While existing research has examined individual models, systematic comparisons of VLMs' visualization literacy remain unexplored. We bridge this gap through a rigorous, first-of-its-kind evaluation of four leading VLMs (GPT-4, Claude, Gemini, and Llama) using standardized assessments: the Visualization Literacy Assessment Test (VLAT) and Critical Thinking Assessment for Literacy in Visualizations (CALVI). Our methodology uniquely combines randomized trials with structured prompting techniques to control for order effects and response variability - a critical consideration overlooked in many VLM evaluations. Our analysis reveals that while specific models demonstrate competence in basic chart interpretation (Claude achieving 67.9% accuracy on VLAT), all models exhibit substantial difficulties in identifying misleading visualization elements (maximum 30.0% accuracy on CALVI). We uncover distinct performance patterns: strong capabilities in interpreting conventional charts like line charts (76-96% accuracy) and detecting hierarchical structures (80-100% accuracy), but consistent difficulties with data-dense visualizations involving multiple encodings (bubble charts: 18.6-61.4%) and anomaly detection (25-30% accuracy). Significantly, we observe distinct uncertainty management behavior across models, with Gemini displaying heightened caution (22.5% question omission) compared to others (7-8%). These findings provide crucial insights for the visualization community by establishing reliable VLM evaluation benchmarks, identifying areas where current models fall short, and highlighting the need for targeted improvements in VLM architectures for visualization tasks.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 68 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

41 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Models31 Oct
  2. 2024How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?24 Jul
  3. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs1 Jun

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
26 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  4. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  5. 2024An Introduction to Vision-Language Modelingcited by 175
  6. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  7. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  8. 2021A Survey of Visual Transformerscited by 117
  9. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  10. 2023ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Languagecited by 71
  11. 2023Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Modelscited by 58
  12. 2022Fairness-aware Adversarial Perturbation Towards Bias Mitigation for Deployed Deep Modelscited by 47
  13. 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
  14. 2024VisEval: A Benchmark for Data Visualization in the Era of Large Language Modelscited by 43
  15. 2024Large language models can be zero-shot anomaly detectors for time series?cited by 30
  16. 2024Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detectioncited by 26
  17. 2023DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articlescited by 24
  18. 2022Misinformed by Visualization: What Do We Learn From Misinformative Visualizations?cited by 19
  19. 2023Mini-VLAT: A Short and Effective Measure of Visualization Literacycited by 16
  20. 2024Are LLMs ready for Visualization?cited by 16
  21. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysiscited by 14
  22. 2023VizAbility: Enhancing Chart Accessibility with LLM-based Conversational Interactioncited by 12
  23. 2024How Aligned are Human Chart Takeaways and LLM Predictions? A Case Study on Bar Charts with Varying Layoutscited by 10
  24. 2024The Limited Impact of Medical Adaptation of Large Language and Vision-Language Modelscited by 9
  25. 2023Adaptive Assessment of Visualization Literacycited by 6
  26. 2022Using Large Language Models to Generate Engaging Captions for Data Visualizationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 10.

20 papers.

Sort:

By area:

  1. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 11
  2. 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Uglycited by 9
  3. 2025DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoningcited by 8
  4. 2025From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualizationcited by 5
  5. 2025Is this chart lying to me? Automating the detection of misleading visualizationscited by 4
  6. 2025Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Modelscited by 2
  7. 2025Tell Me Without Telling Me: Two-Way Prediction of Visualization Literacy and Visual Attentioncited by 1
  8. 2026Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answeringcited by 1
  9. 2026AnnoBench: A Benchmark for Visualization Annotation Generationcited by 1
  10. 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Modelscited by 0
  11. 2025Can AI agents understand spoken conversations about data visualizations in online meetings?cited by 0
  12. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
8 more
  1. 2026ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generationcited by 0
  2. 2026Toward a Machine Bertin: Why Visualization Needs Design Principles for Machine Cognitioncited by 0
  3. 2026Do Large Language Models Understand Data Visualization Principles?cited by 0
  4. 2026De-rendering, Reasoning, and Repairing Charts with Vision-Language Modelscited by 0
  5. 2026Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacycited by 0
  6. 2026More Accurate, Less Human: Gestalt Grouping in Vision Modelscited by 0
  7. 2026LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventionscited by 0
  8. 2026Vibe Analysis: Exploring LLM Adoption by Data Visualization Practitionerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction
  2. 2025The Plot Thickens: Quantitative Part-by-Part Exploration of MLLM Visualization Literacy
  3. 2025Do Vision-Language Models See Visualizations Like Humans? Alignment in Chart Categorization
  4. 2026Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
  5. 2025CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models
  6. 2024Visualization Literacy of Multimodal Large Language Models: A Comparative Study
  7. 2025Evaluating Compliance with Visualization Guidelines in Diagrams for Scientific Publications Using Large Vision Language Models
  8. 2026Do MLLMs See What We See? Analyzing Visualization Literacy Barriers in AI Systems