mlsanityData to 7 Oct 2026

ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning

arXiv 2403.09028 ↗First posted 14 Mar 2024cs.CLcited by 68

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Charts provide visual representations of data and are widely used for analyzing information, addressing queries, and conveying insights to others. Various chart-related downstream tasks have emerged recently, such as question-answering and summarization. A common strategy to solve these tasks is to fine-tune various models originally trained on vision tasks language. However, such task-specific models are not capable of solving a wide range of chart-related tasks, constraining their real-world applicability. To overcome these challenges, we introduce ChartInstruct: a novel chart-specific vision-language Instruction-following dataset comprising 191K instructions generated with 71K charts. We then present two distinct systems for instruction tuning on such datasets: (1) an end-to-end model that connects a vision encoder for chart understanding with a LLM; and (2) a pipeline model that employs a two-step approach to extract chart data tables and input them into the LLM. In experiments on four downstream tasks, we first show the effectiveness of our model--achieving a new set of state-of-the-art results. Further evaluation shows that our instruction-tuning approach supports a wide array of real-world chart comprehension and reasoning scenarios, thereby expanding the scope and applicability of our models to new kinds of tasks.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 60 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

34 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023DOMINO: A Dual-System for Multi-step Visual Language Reasoning4 Oct
  2. 2022DePlot: One-shot visual language reasoning by plot-to-table translation20 Dec
  3. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering19 Dec
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
18 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  3. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  4. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  5. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  6. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  7. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  8. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  9. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  10. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  11. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 77
  12. 2022OpenCQA: Open-ended Question Answering with Chartscited by 60
  13. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  14. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 34
  15. 2020Calliope: Automatic Visual Data Story Generation from a Spreadsheetcited by 30
  16. 2019Searching the Visual Style and Structure of D3 Visualizationscited by 11
  17. 2022Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policiescited by 10
  18. 2023Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarizationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 29; 2026: 18.

68 papers.

Sort:

By area:

  1. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 105
  2. 2024ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Datacited by 92
  3. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  4. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  5. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  6. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  7. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
  8. 2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 32
  9. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  10. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  11. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  12. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
38 more
  1. 2024DataNarrative: Automated Data-Driven Storytelling with Visualizations and Textscited by 25
  2. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problemscited by 24
  3. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 24
  4. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
  5. 2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
  6. 2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Chartscited by 16
  7. 2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
  8. 2025ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understandingcited by 13
  9. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  10. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 11
  11. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 11
  12. 2025Boosting Chart-to-Code Generation in MLLM via Dual Preference-Guided Refinementcited by 10
  13. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  14. 2025Charts Are Not Images: On the Challenges of Scientific Chart Editingcited by 9
  15. 2026QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understandingcited by 9
  16. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 8
  17. 2024FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoningcited by 7
  18. 2025ChartCap: Mitigating Hallucination of Dense Chart Captioningcited by 7
  19. 2025PhysicsMinions: Winning Gold Medals in the Latest Physics Olympiads with a Coevolutionary Multimodal Multi-Agent Systemcited by 7
  20. 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directionscited by 7
  21. 2026SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and Morecited by 7
  22. 2024VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoningcited by 6
  23. 2024Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Modelscited by 6
  24. 2025ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answeringcited by 6
  25. 2024Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directionscited by 5
  26. 2025The Perils of Chart Deception: How Misleading Visualizations Affect Vision-Language Modelscited by 5
  27. 2024ChartAdapter: Large Vision-Language Model for Chart Summarizationcited by 4
  28. 2025START: Spatial and Textual Learning for Chart Understandingcited by 4
  29. 2025ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoningcited by 4
  30. 2024Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trendscited by 3
  31. 2025POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answeringcited by 3
  32. 2025DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboardscited by 3
  33. 2025Do MLLMs Really Understand the Charts?cited by 3
  34. 2025Structured Extraction from Business Process Diagrams Using Vision-Language Modelscited by 3
  35. 2025Chain of Functions: A Programmatic Pipeline for Fine-Grained Chart Reasoning Datacited by 2
  36. 2025In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understandingcited by 2
  37. 2025Chart-CoCa: Self-Improving Chart Understanding of Vision LMs via Code-Driven Synthesis and Candidate-Conditioned Answeringcited by 2
  38. 2025Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Frameworkcited by 2

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning
  2. 2025Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Ugly
  3. 2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?
  4. 2023ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules
  5. 2025ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement
  6. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
  7. 2025ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering
  8. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding