Data to 7 Oct 2026

ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning

arXiv 2402.12185 ↗First posted 19 Feb 2024cs.CVcited by 127

On the map: LLM reasoning → Multimodal reasoning

also in 1 more

Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Recently, many versatile Multi-modal Large Language Models (MLLMs) have emerged continuously. However, their capacity to query information depicted in visual charts and engage in reasoning based on the queried contents remains under-explored. In this paper, to comprehensively and rigorously benchmark the ability of the off-the-shelf MLLMs in the chart domain, we construct ChartX, a multi-modal evaluation set covering 18 chart types, 7 chart tasks, 22 disciplinary topics, and high-quality chart data. Besides, we develop ChartVLM to offer a new perspective on handling multi-modal tasks that strongly depend on interpretable patterns, such as reasoning tasks in the field of charts or geometric images. We evaluate the chart-related ability of mainstream MLLMs and our ChartVLM on the proposed ChartX evaluation set. Extensive experiments demonstrate that ChartVLM surpasses both versatile and chart-related large models, achieving results comparable to GPT-4V. We believe that our study can pave the way for further exploration in creating a more comprehensive chart evaluation set and developing more interpretable multi-modal models. Both ChartX and ChartVLM are available at: https://github.com/Alpha-Innovator/ChartVLM

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 59 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

37 of 59 references.

Closest ancestors · 4

Same direction, newest first

Other references · 33

2020Language Models are Few-Shot Learnerscited by 32,199
2023GPT-4 Technical Reportcited by 18,132
2022Training language models to follow instructions with human feedbackcited by 15,712
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023Improved Baselines with Visual Instruction Tuningcited by 4,362
2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
2023CogAgent: A Visual Language Model for GUI Agentscited by 692
2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610

21 more

2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
2019PlotQA: Reasoning over Scientific Plotscited by 317
2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 278
2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 77
2023ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Languagecited by 71
2022OpenCQA: Open-ended Question Answering with Chartscited by 60
2023TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPTcited by 57
2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
2023Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queriescited by 38
2023ChartCheck: Explainable Fact-Checking over Real-World Chart Imagescited by 25
2023StructChart: On the Schema, Metric, and Augmentation for Visual Chart Understandingcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 71; 2026: 29.

127 papers.

Sort:

By area:

2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 144
2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 54
2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52

38 more

2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answeringcited by 50
2025SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writingcited by 47
2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
2024Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matchingcited by 39
2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 35
2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 32
2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 31
2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Modelscited by 30
2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 29
2025ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editingcited by 28
2024FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowchartscited by 27
2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 24
2025EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewardscited by 24
2024GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-trainingcited by 23
2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
2025OmniCaptioner: One Captioner to Rule Them Allcited by 19
2025AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMscited by 18
2025Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generationcited by 18
2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustnesscited by 17
2024Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedbackcited by 16
2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Chartscited by 16
2025ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learningcited by 15
2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
2024PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Samplingcited by 14
2025KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understandingcited by 14
2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
2025ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answeringcited by 11

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way