mlsanityData to 14 Oct 2026

Chart-to-Text: A Large-Scale Benchmark for Chart Summarization

arXiv 2203.06486 ↗First posted 12 Mar 2022cs.CLcited by 169

Not on the map. Why?

Cited by 70 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Charts are commonly used for exploring data and communicating insights. Generating natural language summaries from charts can be very helpful for people in inferring key insights that would otherwise require a lot of cognitive and perceptual efforts. We present Chart-to-text, a large-scale benchmark with two datasets and a total of 44,096 charts covering a wide range of topics and chart types. We explain the dataset construction process and analyze the datasets. We also introduce a number of state-of-the-art neural models as baselines that utilize image captioning and data-to-text generation techniques to tackle two problem variations: one assumes the underlying data table of the chart is available while the other needs to extract data from chart images. Our analysis with automatic and human evaluation shows that while our best models usually generate fluent summaries and yield reasonable BLEU scores, they also suffer from hallucinations and factual errors as well as difficulties in correctly explaining complex patterns and trends in charts.

Key facts. Cited by 169 papers in the map (data through 14 Oct 2026). Builds on 31 of its 60 references that are in the map. First posted on arXiv 12 Mar 2022.

Builds on

31 of 60 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
19 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  2. 2018nocaps: novel object captioning at scalecited by 444
  3. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  4. 2019Character Region Awareness for Text Detectioncited by 278
  5. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 268
  6. 2017Challenges in Data-to-Document Generationcited by 226
  7. 2019What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysiscited by 191
  8. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  9. 2020Logical Natural Language Generation from Open-Domain Tablescited by 90
  10. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 78
  11. 2021SciCap: Generating Captions for Scientific Figurescited by 77
  12. 2021Plan-then-Generate: Controlled Data-to-Text Generation via Planningcited by 52
  13. 2020Logic2Text: High-Fidelity Natural Language Generation from Logical Formscited by 45
  14. 2021Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Contentcited by 39
  15. 2015What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignmentcited by 31
  16. 2019Figure Captioning with Reasoning and Sequence-Level Trainingcited by 21
  17. 2018DataSite: Proactive Visual Data Exploration with Computation of Insight-based Recommendationscited by 12
  18. 2021Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Chartscited by 12
  19. 2016Reference-Aware Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 23; 2024: 41; 2025: 59; 2026: 39.

169 papers.

Sort:

By area:

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  4. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  5. 2023Data-centric Artificial Intelligence: A Surveycited by 203
  6. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  7. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 163
  8. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 160
  9. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 160
  10. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
  11. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 133
  12. 2024ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generationcited by 129
38 more
  1. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 128
  2. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  3. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  4. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 112
  5. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  6. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 108
  7. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generationcited by 92
  8. 2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelcited by 89
  9. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  10. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  11. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  12. 2023Is GPT-4 a Good Data Analyst?cited by 70
  13. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  14. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  15. 2022OpenCQA: Open-ended Question Answering with Chartscited by 61
  16. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 55
  17. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 53
  18. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversioncited by 53
  19. 2025FineVision: Open Data Is All You Needcited by 53
  20. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 49
  21. 2024VisEval: A Benchmark for Data Visualization in the Era of Large Language Modelscited by 45
  22. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  23. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 35
  24. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  25. 2023ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summariescited by 32
  26. 2023mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Modelcited by 32
  27. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  28. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  29. 2022Chart Question Answering: State of the Art and Future Directionscited by 26
  30. 2023ChartCheck: Explainable Fact-Checking over Real-World Chart Imagescited by 26
  31. 2024DataNarrative: Automated Data-Driven Storytelling with Visualizations and Textscited by 25
  32. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  33. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  34. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 25
  35. 2024Generative AI for Visualization: State of the Art and Future Directionscited by 23
  36. 2023GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captionscited by 21
  37. 2024Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQAcited by 19
  38. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19

50 most cited of 169 All 169 (JSON)