# Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model

arXiv: https://arxiv.org/abs/2010.09142 · HTML: https://mlsanity.com/paper/2010.09142/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2020.

## Built on by (most cited first, 30 of 78)

- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale](https://mlsanity.com/paper/2412.05237/) (2024)
- [NVLM: Open Frontier-Class Multimodal LLMs](https://mlsanity.com/paper/2409.11402/) (2024)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning](https://mlsanity.com/paper/2409.20566/) (2024)
- [ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2403.09028/) (2024)
- [OpenCQA: Open-ended Question Answering with Charts](https://mlsanity.com/paper/2210.06628/) (2022)
- [From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models](https://mlsanity.com/paper/2403.12027/) (2024)
- [TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning](https://mlsanity.com/paper/2404.16635/) (2024)
- [FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities](https://mlsanity.com/paper/2505.20147/) (2025)
- [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021)
- [OneChart: Purify the Chart Structural Extraction via One Auxiliary Token](https://mlsanity.com/paper/2404.09987/) (2024)
- [SOLO: A Single Transformer for Scalable Vision-Language Modeling](https://mlsanity.com/paper/2407.06438/) (2024)
- [Language Models as Inductive Reasoners](https://mlsanity.com/paper/2212.10923/) (2022)
- [ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries](https://mlsanity.com/paper/2304.13620/) (2023)
- [SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding](https://mlsanity.com/paper/2503.18943/) (2025)
- [Chart Question Answering: State of the Art and Future Directions](https://mlsanity.com/paper/2205.03966/) (2022)
- [ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules](https://mlsanity.com/paper/2304.02173/) (2023)
- [DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articles](https://mlsanity.com/paper/2308.04076/) (2023)
- [Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence](https://mlsanity.com/paper/2502.09927/) (2025)
- [Underreporting of errors in NLG output, and what to do about it](https://mlsanity.com/paper/2108.01182/) (2021)
- [Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers](https://mlsanity.com/paper/2108.00308/) (2021)
- [LineCap: Line Charts for Data Visualization Captioning Models](https://mlsanity.com/paper/2207.07243/) (2022)

## Abstract (arXiv)

Information visualizations such as bar charts and line charts are very popular for exploring data and communicating insights. Interpreting and making sense of such visualizations can be challenging for some people, such as those who are visually impaired or have low visualization literacy. In this work, we introduce a new dataset and present a neural model for automatically generating natural language summaries for charts. The generated summaries provide an interpretation of the chart and convey the key insights found within that chart. Our neural model is developed by extending the state-of-the-art model for the data-to-text generation task, which utilizes a transformer-based encoder-decoder architecture. We found that our approach outperforms the base model on a content selection metric by a wide margin (55.42% vs. 8.49%) and generates more informative, concise, and coherent summaries.