mlsanityData to 7 Oct 2026

ChartLlama: A Multimodal LLM for Chart Understanding and Generation

arXiv 2311.16483 ↗First posted 27 Nov 2023cs.CV cs.CLcited by 158

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Multi-modal large language models have demonstrated impressive performances on most vision-language tasks. However, the model generally lacks the understanding capabilities for specific domain data, particularly when it comes to interpreting chart figures. This is mainly due to the lack of relevant multi-modal instruction tuning datasets. In this article, we create a high-quality instruction-tuning dataset leveraging GPT-4. We develop a multi-step data generation process in which different steps are responsible for generating tabular data, creating chart figures, and designing instruction tuning data separately. Our method's flexibility enables us to generate diverse, high-quality instruction-tuning data consistently and efficiently while maintaining a low resource expenditure. Additionally, it allows us to incorporate a wider variety of chart and task types not yet featured in existing datasets. Next, we introduce ChartLlama, a multi-modal large language model that we've trained using our created dataset. ChartLlama outperforms all prior methods in ChartQA, Chart-to-text, and Chart-extraction evaluation benchmarks. Additionally, ChartLlama significantly improves upon the baseline in our specially compiled chart dataset, which includes new chart and task types. The results of ChartLlama confirm the value and huge potential of our proposed data generation method in enhancing chart comprehension.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 36 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

28 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning24 May
  2. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering19 Dec

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
14 more
  1. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  3. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  4. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  6. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  7. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  8. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  9. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  10. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  11. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  12. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  13. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  14. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 51; 2025: 69; 2026: 35.

158 papers.

Sort:

By area:

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  2. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  3. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  4. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  5. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  6. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  7. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  8. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  9. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  10. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  11. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  12. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
38 more
  1. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 81
  2. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  3. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  4. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  5. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilitiescited by 57
  6. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 54
  7. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  8. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  9. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
  10. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answeringcited by 50
  11. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  12. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
  13. 2024VisEval: A Benchmark for Data Visualization in the Era of Large Language Modelscited by 43
  14. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  15. 2025METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scalingcited by 33
  16. 2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 32
  17. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  18. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  19. 2024SketchAgent: Language-Driven Sequential Sketch Generationcited by 30
  20. 2024FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowchartscited by 27
  21. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  22. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 24
  23. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  24. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
  25. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  26. 2024Chart-based Reasoning: Transferring Capabilities from LLMs to VLMscited by 20
  27. 2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
  28. 2024SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language modelscited by 18
  29. 2025Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generationcited by 18
  30. 2024Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedbackcited by 16
  31. 2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Chartscited by 16
  32. 2025ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learningcited by 15
  33. 2025LLM/Agent-as-Data-Analyst: A Surveycited by 15
  34. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  35. 2024ChartReformer: Natural Language-Driven Chart Image Editingcited by 13
  36. 2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
  37. 2025InterChat: Enhancing Generative Visual Analytics using Multimodal Interactionscited by 13
  38. 2025ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understandingcited by 13

50 most cited of 158 All 158 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning
  2. 2025In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding
  3. 2026ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation
  4. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding
  5. 2026ChartComplete: A Taxonomy-based Inclusive Chart Dataset
  6. 2025Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction
  7. 2025From Charts to Code: A Hierarchical Benchmark for Multimodal Models
  8. 2026Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework