mlsanityData to 13 Oct 2026

MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning

arXiv 2311.10774 ↗First posted 15 Nov 2023cs.CL cs.AIcited by 149

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

With the rapid development of large language models (LLMs) and their integration into large multimodal models (LMMs), there has been impressive progress in zero-shot completion of user-oriented vision-language tasks. However, a gap remains in the domain of chart image understanding due to the distinct abstract components in charts. To address this, we introduce a large-scale MultiModal Chart Instruction (MMC-Instruction) dataset comprising 600k instances supporting diverse tasks and chart types. Leveraging this data, we develop MultiModal Chart Assistant (MMCA), an LMM that achieves state-of-the-art performance on existing chart QA benchmarks. Recognizing the need for a comprehensive evaluation of LMM chart understanding, we also propose a MultiModal Chart Benchmark (MMC-Benchmark), a comprehensive human-annotated benchmark with nine distinct tasks evaluating reasoning capabilities over charts. Extensive experiments on MMC-Benchmark reveal the limitations of existing LMMs on correctly interpreting charts, even for the most recent GPT-4V model. Our work provides an instruction-tuning methodology and benchmark to advance multimodal understanding of charts. Code and data are available at https://github.com/FuxiaoLiu/MMC.

Key facts. Cited by 149 papers in the map (data through 13 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 45 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

30 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning24 May

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2023Segment Anythingcited by 8,195
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,461
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,295
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,230
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 898
  12. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 680
17 more
  1. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 663
  2. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 515
  3. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 362
  4. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 339
  5. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  6. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 316
  7. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  8. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 247
  9. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  10. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 111
  11. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  12. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 79
  13. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  14. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 67
  15. 2019LEAF-QA: Locate, Encode & Attend for Figure Question Answeringcited by 53
  16. 2023Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Leadcited by 40
  17. 2023DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documentscited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 64; 2025: 61; 2026: 23.

149 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,586
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,041
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 510
  4. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  5. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 323
  6. 2024NVILA: Efficient Frontier Visual Language Modelscited by 208
  7. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  8. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 169
  9. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 157
  10. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 135
  11. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  12. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
38 more
  1. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 118
  2. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  3. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 89
  4. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
  5. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  6. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  7. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  8. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 74
  9. 2024TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domainscited by 69
  10. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  11. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  12. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Datacited by 67
  13. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  14. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  15. 2025FineVision: Open Data Is All You Needcited by 53
  16. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  17. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  18. 2024ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answeringcited by 50
  19. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 48
  20. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  21. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 45
  22. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  23. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  24. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  25. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  26. 2024Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statementscited by 34
  27. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  28. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  29. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  30. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 31
  31. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  32. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  33. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluationcited by 29
  34. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  35. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  36. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  37. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platformscited by 26
  38. 2024VILA2: VILA Augmented VILAcited by 25

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generation
  2. 2025From Charts to Code: A Hierarchical Benchmark for Multimodal Models
  3. 2025ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answering
  4. 2025Do MLLMs Really Understand the Charts?
  5. 2026Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework
  6. 2025ChartM3: Benchmarking Chart Editing with Multimodal Instructions
  7. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
  8. 2025Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts