mlsanityData to 14 Oct 2026

ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering

arXiv 2504.05506 ↗First posted 7 Apr 2025cs.CLcited by 108

Not on the map. Why?

Cited by 57 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Charts are ubiquitous, as people often use them to analyze data, answer questions, and discover critical insights. However, performing complex analytical tasks with charts requires significant perceptual and cognitive effort. Chart Question Answering (CQA) systems automate this process by enabling models to interpret and reason with visual representations of data. However, existing benchmarks like ChartQA lack real-world diversity and have recently shown performance saturation with modern large vision-language models (LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark that includes 1,341 charts from 157 diverse sources, spanning various chart types, including infographics and dashboards, and featuring 1,948 questions in various types, such as multiple-choice, conversational, hypothetical, and unanswerable questions, to better reflect real-world challenges. Our evaluations with 21 models show a substantial performance drop for LVLMs on ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on ChartQAPro, underscoring the complexity of chart reasoning. We complement our findings with detailed error analyses and ablation studies, identifying key challenges and opportunities for advancing LVLMs in chart understanding and reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

26 of 42 references.

References · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
14 more
  1. 2018CoQA: A Conversational Question Answering Challengecited by 683
  2. 2019Scene Text Visual Question Answeringcited by 331
  3. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  4. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  5. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
  6. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  7. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 160
  8. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  9. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  10. 2022OpenCQA: Open-ended Question Answering with Chartscited by 61
  11. 2019LEAF-QA: Locate, Encode & Attend for Figure Question Answeringcited by 54
  12. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 35
  13. 2024DataNarrative: Automated Data-Driven Storytelling with Visualizations and Textscited by 25
  14. 2019Searching the Visual Style and Structure of D3 Visualizationscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 36; 2026: 72.

108 papers.

Sort:

By area:

  1. 2025Ovis2.5 Technical Reportcited by 108
  2. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Modelscited by 32
  3. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 30
  4. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 25
  5. 2026Qianfan-OCR: A Unified End-to-End Model for Document Intelligencecited by 21
  6. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 18
  7. 2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Chartscited by 17
  8. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoningcited by 16
  9. 2025SciTS: Scientific Time Series Understanding and Generation with LLMscited by 15
  10. 2025Thinking with Programming Vision: Towards a Unified View for Thinking with Imagescited by 15
  11. 2025ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answeringcited by 13
  12. 2025Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curationcited by 13
38 more
  1. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 12
  2. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 10
  3. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 10
  4. 2025Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Inputcited by 9
  5. 2025VACoT: Rethinking Visual Data Augmentation with VLMscited by 9
  6. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 8
  7. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understandingcited by 8
  8. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 8
  9. 2025VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMscited by 7
  10. 2025Qianfan-VL: Domain-Enhanced Universal Vision-Language Modelscited by 7
  11. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 6
  12. 2026Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoningcited by 6
  13. 2026SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memorycited by 6
  14. 2025Describe Anything Model for Visual Question Answering on Text-rich Imagescited by 5
  15. 2025ChartHal: A Fine-grained Framework Evaluating Hallucination of Large Vision Language Models in Chart Understandingcited by 5
  16. 2025Losing the Plot: How VLM responses degrade on imperfect chartscited by 5
  17. 2025InfoDet: A Dataset for Infographic Element Detectioncited by 4
  18. 2025InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Informationcited by 4
  19. 2025START: Spatial and Textual Learning for Chart Understandingcited by 4
  20. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 4
  21. 2026MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judgecited by 4
  22. 2026Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillationcited by 4
  23. 2026PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Modelscited by 4
  24. 2026Intern-S2-Preview: Scientific Agentic Foundation Modelcited by 4
  25. 2025POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answeringcited by 3
  26. 2025DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboardscited by 3
  27. 2025Do MLLMs Really Understand the Charts?cited by 3
  28. 2025Composition-Grounded Data Synthesis for Visual Reasoningcited by 3
  29. 2025Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Modelscited by 3
  30. 2025ChartAB: A Benchmark for Chart Grounding & Dense Alignmentcited by 3
  31. 2025See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoningcited by 3
  32. 2026AdaptMMBench: Benchmarking Adaptive Multimodal Reasoning for Mode Selection and Reasoning Processcited by 3
  33. 2026TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RLcited by 3
  34. 2026Scalable Visual Pretraining for Language Intelligencecited by 3
  35. 2026MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agentscited by 3
  36. 2025ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinementcited by 2
  37. 2026SciMDR: Advancing Scientific Multimodal Document Reasoningcited by 2
  38. 2026ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Chartscited by 2

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Chart Question Answering from Real-World Analytical Narratives
  2. 2025ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering
  3. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering
  4. 2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness
  5. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems
  6. 2023DCQA: Document-Level Chart Question Answering towards Complex Reasoning and Common-Sense Understanding
  7. 2024Charting the Future: Using Chart Question-Answering for Scalable Evaluation of LLM-Driven Data Visualizations
  8. 2025Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts