mlsanityData to 7 Oct 2026

Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness

arXiv 2407.11229 ↗First posted 15 Jul 2024cs.CL cs.AI cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

Chart question answering (CQA) is a crucial area of Visual Language Understanding. However, the robustness and consistency of current Visual Language Models (VLMs) in this field remain under-explored. This paper evaluates state-of-the-art VLMs on comprehensive datasets, developed specifically for this study, encompassing diverse question categories and chart formats. We investigate two key aspects: 1) the models' ability to handle varying levels of chart and question complexity, and 2) their robustness across different visual representations of the same underlying data. Our analysis reveals significant performance variations based on question and chart types, highlighting both strengths and weaknesses of current models. Additionally, we identify areas for improvement and propose future research directions to build more robust and reliable CQA systems. This study sheds light on the limitations of current models and paves the way for future advancements in the field.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

22 of 35 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  6. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  7. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  8. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  9. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  10. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  11. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  12. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
10 more
  1. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  2. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  3. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  4. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  5. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  6. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  7. 2022OpenCQA: Open-ended Question Answering with Chartscited by 60
  8. 2019LEAF-QA: Locate, Encode & Attend for Figure Question Answeringcited by 53
  9. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  10. 2024Enhancing Question Answering on Charts Through Effective Pre-training Taskscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 9.

17 papers.

Sort:

By area:

  1. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 54
  2. 2025MapIQ: Evaluating Multimodal Large Language Models for Map Question Answeringcited by 5
  3. 2025Does visualization help AI understand data?cited by 3
  4. 2025Do MLLMs Really Understand the Charts?cited by 3
  5. 2025Does It Run and Is That Enough? Revisiting Text-to-Chart Generation with a Multi-Agent Approachcited by 1
  6. 2025ViStruct: Simulating Expert-Like Reasoning Through Task Decomposition and Visual Attention Cuescited by 1
  7. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  8. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
  9. 2025Seeing isn't Hearing: Benchmarking Vision Language Models at Interpreting Spectrogramscited by 0
  10. 2026Beyond Single Plots: A Benchmark for Question Answering on Multi-Chartscited by 0
  11. 2026Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translationcited by 0
  12. 2026Unlocking UML Class Diagram Understanding in Vision Language Modelscited by 0
5 more
  1. 2026VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Followingcited by 0
  2. 2026MMClima: A Framework for Multimodal Climate Science Data and Evaluationcited by 0
  3. 2026EvidenceLens: A Claim-Evidence Matrix for Auditing Financial Question Answeringcited by 0
  4. 2026When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Renderingcited by 0
  5. 2026Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Readingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering
  2. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering
  3. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning
  4. 2026What Lies Beneath: A Call for Distribution-based Visual Question & Answer Datasets
  5. 2025Chart Question Answering from Real-World Analytical Narratives
  6. 2024Charting the Future: Using Chart Question-Answering for Scalable Evaluation of LLM-Driven Data Visualizations
  7. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems
  8. 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Ugly