mlsanityData to 7 Oct 2026

CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs

arXiv 2406.18521 ↗First posted 26 Jun 2024cs.CL cs.CVcited by 229

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Chart understanding plays a pivotal role when applying Multimodal Large Language Models (MLLMs) to real-world tasks such as analyzing scientific papers or financial reports. However, existing datasets often focus on oversimplified and homogeneous charts with template-based questions, leading to an over-optimistic measure of progress. We demonstrate that although open-source models can appear to outperform strong proprietary models on these benchmarks, a simple stress test with slightly different charts or questions can deteriorate performance by up to 34.5%. In this work, we propose CharXiv, a comprehensive evaluation suite involving 2,323 natural, challenging, and diverse charts from arXiv papers. CharXiv includes two types of questions: 1) descriptive questions about examining basic chart elements and 2) reasoning questions that require synthesizing information across complex visual elements in the chart. To ensure quality, all charts and questions are handpicked, curated, and verified by human experts. Our results reveal a substantial, previously underestimated gap between the reasoning skills of the strongest proprietary model (i.e., GPT-4o), which achieves 47.1% accuracy, and the strongest open-source model (i.e., InternVL Chat V1.5), which achieves 29.2%. All models lag far behind human performance of 80.5%, underscoring weaknesses in the chart understanding capabilities of existing MLLMs. We hope CharXiv facilitates future research on MLLM chart understanding by providing a more realistic and faithful measure of progress. Project page and leaderboard: https://charxiv.github.io/

Key facts. Cited by 229 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 71 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

31 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning19 Feb
  2. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts26 Dec

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Mistral 7Bcited by 3,065
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  11. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
17 more
  1. 2024Yi: Open Foundation Models by 01.AIcited by 706
  2. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  3. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  4. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  5. 2024What matters when building vision-language models?cited by 321
  6. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  7. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  8. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  9. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  10. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  11. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  12. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  13. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  14. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  15. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  16. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  17. 2024Improving Language Understanding from Screenshotscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 105; 2026: 106.

229 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  7. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  8. 2026Qwen3.5-Omni Technical Reportcited by 146
  9. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  10. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  11. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  12. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
38 more
  1. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 114
  2. 2025MiMo-VL Technical Reportcited by 106
  3. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 105
  4. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  5. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  6. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 81
  7. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  8. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  9. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  10. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Modelscited by 56
  11. 2025Kwai Keye-VL Technical Reportcited by 47
  12. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  13. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  14. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  15. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  16. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  17. 2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
  18. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  19. 2025TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoningcited by 34
  20. 2026MiniMax Sparse Attentioncited by 34
  21. 2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 32
  22. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  23. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  24. 2025Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoningcited by 30
  25. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  26. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 29
  27. 2025ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editingcited by 28
  28. 2026STEP3-VL-10B Technical Reportcited by 28
  29. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  30. 2024M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Modelscited by 26
  31. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  32. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  33. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  34. 2026Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencecited by 25
  35. 2024AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMMcited by 24
  36. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 24
  37. 2025Simple o3: Towards Interleaved Vision-Language Reasoningcited by 24
  38. 2025CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimizationcited by 24

50 most cited of 229 All 229 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information
  2. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering
  3. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems
  4. 2026LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning
  5. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs
  6. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning
  7. 2025FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models
  8. 2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?