mlsanityData to 7 Oct 2026

ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning

arXiv 2401.02384 ↗First posted 4 Jan 2024cs.CVcited by 79

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Charts play a vital role in data visualization, understanding data patterns, and informed decision-making. However, their unique combination of graphical elements (e.g., bars, lines) and textual components (e.g., labels, legends) poses challenges for general-purpose multimodal models. While vision-language models trained on chart data excel in comprehension, they struggle with generalization. To address these challenges, we propose ChartAssistant, a chart-based vision-language model for universal chart comprehension and reasoning. ChartAssistant leverages ChartSFT, a comprehensive dataset covering diverse chart-related tasks with basic (e.g. bars and pies) and specialized (e.g. radars, and bubbles) chart types. It undergoes a two-stage training process, starting with pre-training on chart-to-table parsing to align chart and text, followed by multitask instruction-following fine-tuning. This approach enables ChartAssistant to achieve competitive performance across various chart tasks. Experimental results demonstrate significant performance gains over the state-of-the-art UniChart and Chartllama method, especially outperforming them on real-world chart data with zero-shot setting. The code and data are available at https://github.com/OpenGVLab/ChartAst.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 66 references that are in the map. First posted on arXiv 4 Jan 2024.

Builds on

44 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generation27 Nov
  2. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning15 Nov
  3. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning24 May
  4. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering19 Dec

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
28 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  4. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  6. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  7. 2021OCR-free Document Understanding Transformercited by 400
  8. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  9. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  10. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  11. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  12. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  13. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  14. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  15. 2019Image Captioning: Transforming Objects into Wordscited by 171
  16. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  17. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  18. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  19. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 111
  20. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  21. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 79
  22. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  23. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  24. 2022OpenCQA: Open-ended Question Answering with Chartscited by 60
  25. 2019Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehensioncited by 27
  26. 2022Chart Question Answering: State of the Art and Future Directionscited by 26
  27. 2023Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Modelscited by 20
  28. 2023Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 33; 2026: 13.

79 papers.

Sort:

By area:

  1. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  2. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  3. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  4. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 118
  5. 2024ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Datacited by 92
  6. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 88
  7. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  8. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  9. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  10. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 51
  11. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answeringcited by 50
  12. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
38 more
  1. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  2. 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
  3. 2024EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understandingcited by 32
  4. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  5. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  6. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  7. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  8. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
  9. 2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
  10. 2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustnesscited by 17
  11. 2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Chartscited by 16
  12. 2025ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learningcited by 15
  13. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  14. 2024EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmarkcited by 14
  15. 2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
  16. 2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
  17. 2025ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understandingcited by 13
  18. 2024StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyondcited by 11
  19. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 11
  20. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  21. 2024mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoningcited by 9
  22. 2025ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Datasetcited by 9
  23. 2025Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptationcited by 9
  24. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 8
  25. 2024SIMPLOT: Enhancing Chart Question Answering by Distilling Essentialscited by 7
  26. 2024MMFactory: A Universal Solution Search Engine for Vision-Language Taskscited by 7
  27. 2025ChartCap: Mitigating Hallucination of Dense Chart Captioningcited by 7
  28. 2026SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and Morecited by 7
  29. 2024VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoningcited by 6
  30. 2024Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Modelscited by 6
  31. 2025Evaluating LLMs for Visualization Generation and Understandingcited by 5
  32. 2025Losing the Plot: How VLM responses degrade on imperfect chartscited by 5
  33. 2026Visual Self-Refine: A Pixel-Guided Paradigm for Accurate Chart Parsingcited by 5
  34. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understandingcited by 5
  35. 2024ChartAdapter: Large Vision-Language Model for Chart Summarizationcited by 4
  36. 2025Towards Understanding Graphical Perception in Large Multimodal Modelscited by 4
  37. 2025Visual Programmability: A Guide for Code-as-Thought in Chart Understandingcited by 4
  38. 2024AltChart: Enhancing VLM-based Chart Summarization Through Multi-Pretext Taskscited by 3

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts
  2. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning
  3. 2026Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding
  4. 2023ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules
  5. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts
  6. 2025START: Spatial and Textual Learning for Chart Understanding
  7. 2025ChartAnchor: Chart Grounding with Structural-Semantic Fidelity
  8. 2026CharTool: Tool-Integrated Visual Reasoning for Chart Understanding