mlsanityData to 7 Oct 2026

ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering

arXiv 2405.07001 ↗First posted 11 May 2024cs.CL cs.AI cs.CVcited by 50

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Chart question answering (ChartQA) tasks play a critical role in interpreting and extracting insights from visualization charts. While recent advancements in multimodal large language models (MLLMs) like GPT-4o have shown promise in high-level ChartQA tasks, such as chart captioning, their effectiveness in low-level ChartQA tasks (e.g., identifying correlations) remains underexplored. In this paper, we address this gap by evaluating MLLMs on low-level ChartQA using a newly curated dataset, ChartInsights, which consists of 22,347 (chart, task, query, answer) covering 10 data analysis tasks across 7 chart types. We systematically evaluate 19 advanced MLLMs, including 12 open-source and 7 closed-source models. The average accuracy rate across these models is 39.8%, with GPT-4o achieving the highest accuracy at 69.17%. To further explore the limitations of MLLMs in low-level ChartQA, we conduct experiments that alter visual elements of charts (e.g., changing color schemes, adding image noise) to assess their impact on the task effectiveness. Furthermore, we propose a new textual prompt strategy, Chain-of-Charts, tailored for low-level ChartQA tasks, which boosts performance by 14.41%, achieving an accuracy of 83.58%. Finally, incorporating a visual prompt strategy that directs attention to relevant visual elements further improves accuracy to 84.32%.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 67 references that are in the map. First posted on arXiv 11 May 2024.

Builds on

33 of 67 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning19 Feb
  2. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning4 Jan
  3. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts26 Dec
  4. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning15 Dec
  5. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generation27 Nov
  6. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning15 Nov
  7. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning24 May

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  12. 2019PlotQA: Reasoning over Scientific Plotscited by 317
14 more
  1. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  2. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  3. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  4. 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 63
  5. 2021Towards Natural Language Interfaces for Data Visualization: A Surveycited by 58
  6. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  7. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  8. 2023Do Large Language Models Know about Facts?cited by 49
  9. 2021nvBench: A Large-Scale Synthesized Dataset for Cross-Domain Natural Language to Visualization Taskcited by 27
  10. 2023ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rulescited by 24
  11. 2024Generative AI for Visualization: State of the Art and Future Directionscited by 23
  12. 2022Towards Natural Language-Based Visualization Authoringcited by 22
  13. 2023Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairscited by 15
  14. 2021A Review and Collation of Graphical Perception Knowledge for Visualization Recommendationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 27; 2026: 23.

50 papers.

Sort:

By area:

  1. 2025Self-Supervised Prompt Optimizationcited by 37
  2. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  3. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 20
  4. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoningcited by 18
  5. 2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Chartscited by 16
  6. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  7. 2025Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answeringcited by 14
  8. 2026IGenBench: Benchmarking the Reliability of Text-to-Infographic Generationcited by 14
  9. 2025EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routingcited by 13
  10. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  11. 2025InteractComp: Evaluating Search Agents With Ambiguous Queriescited by 12
  12. 2025Data-to-Dashboard: Multi-Agent LLM Framework for Insightful Visualization in Enterprise Analyticscited by 11
38 more
  1. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 11
  2. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  3. 2025See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMscited by 7
  4. 2025VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanismcited by 7
  5. 2025LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understandingcited by 7
  6. 2025ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answeringcited by 6
  7. 2025ChartMark: A Structured Grammar for Chart Annotationcited by 6
  8. 2025ChartCards: A Chart-Metadata Generation Framework for Multi-Task Chart Understandingcited by 5
  9. 2025CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Modelscited by 5
  10. 2025MapIQ: Evaluating Multimodal Large Language Models for Map Question Answeringcited by 5
  11. 2025PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistenciescited by 4
  12. 2025Arrow-Guided VLM: Enhancing Flowchart Understanding via Arrow Direction Encodingcited by 3
  13. 2026MultiVis-Agent: A Multi-Agent Framework with Logic Rules for Reliable and Comprehensive Cross-Modal Data Visualizationcited by 3
  14. 2026Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewardscited by 3
  15. 2026DataMagic: Transforming Tabular Data into Data Insight Videocited by 2
  16. 2026Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Frameworkcited by 2
  17. 2025Boosting Text-to-Chart Retrieval through Training with Synthesized Semantic Insightscited by 1
  18. 2025FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Modelscited by 1
  19. 2025ClimateSOM: A Visual Analysis Workflow for Climate Ensemble Datasetscited by 1
  20. 2025ChartM3: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehensioncited by 1
  21. 2026UPA: Unsupervised Prompt Agent via Tree-Based Search and Selectioncited by 1
  22. 2026Beyond Description: A Multimodal Agent Framework for Insightful Chart Summarizationcited by 1
  23. 2026Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answeringcited by 1
  24. 2026Generating Statistical Charts with Validation-Driven LLM Workflowscited by 1
  25. 2026TuneAhead: Predicting Fine-tuning Performance Before Full Training Beginscited by 1
  26. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
  27. 2025CycleChart: A Unified Consistency-Based Learning Framework for Bidirectional Chart Understanding and Generationcited by 0
  28. 2026Toward a Machine Bertin: Why Visualization Needs Design Principles for Machine Cognitioncited by 0
  29. 2026Evaluating Prompting Strategies for Chart Question Answering with Large Language Modelscited by 0
  30. 2026Beyond Single Plots: A Benchmark for Question Answering on Multi-Chartscited by 0
  31. 2026Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Modelscited by 0
  32. 2026Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacycited by 0
  33. 2026A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Predictioncited by 0
  34. 2026UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generationcited by 0
  35. 2026Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Videocited by 0
  36. 2026A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Imagescited by 0
  37. 2026DVBench: Benchmarking MLLMs for Understanding Dynamic Charts and Narratives in Data Videoscited by 0
  38. 2026LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning
  2. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems
  3. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering
  4. 2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness
  5. 2025POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering
  6. 2025In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding
  7. 2025Chart Question Answering from Real-World Analytical Narratives
  8. 2024mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning