# ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries

arXiv: https://arxiv.org/abs/2304.13620 · HTML: https://mlsanity.com/paper/2304.13620/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [OpenCQA: Open-ended Question Answering with Charts](https://mlsanity.com/paper/2210.06628/) (2022)
- [Chart Question Answering: State of the Art and Future Directions](https://mlsanity.com/paper/2205.03966/) (2022)
- [WSL-DS: Weakly Supervised Learning with Distant Supervision for Query Focused Multi-Document Abstractive Summarization](https://mlsanity.com/paper/2011.01421/) (2020)
- [Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization](https://mlsanity.com/paper/2112.11670/) (2021)
- [AutoChart: A Dataset for Chart-to-Text Generation Task](https://mlsanity.com/paper/2108.06897/) (2021)
- [BLINK with Elasticsearch for Efficient Entity Linking in Business Conversations](https://mlsanity.com/paper/2205.04438/) (2022)
- [Reverse-engineering Bar Charts Using Neural Networks](https://mlsanity.com/paper/2009.02491/) (2020)
- [Efficient Inference For Neural Machine Translation](https://mlsanity.com/paper/2010.02416/) (2020)
- [Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop](https://mlsanity.com/paper/2211.01354/) (2022)
- [Data-to-text Generation by Splicing Together Nearest Neighbors](https://mlsanity.com/paper/2101.08248/) (2021)
- [Tensor Fields for Data Extraction from Chart Images: Bar Charts and Scatter Plots](https://mlsanity.com/paper/2010.02319/) (2020)

## Built on by (most cited first, 30 of 32)

- [ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation](https://mlsanity.com/paper/2406.09961/) (2024)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [ChartBench: A Benchmark for Complex Visual Reasoning in Charts](https://mlsanity.com/paper/2312.15915/) (2023)
- [A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations](https://mlsanity.com/paper/2407.04069/) (2024)
- [From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models](https://mlsanity.com/paper/2403.12027/) (2024)
- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning](https://mlsanity.com/paper/2407.20174/) (2024)
- [DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts](https://mlsanity.com/paper/2408.05346/) (2024)
- [Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA](https://mlsanity.com/paper/2403.16385/) (2024)
- [Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering](https://mlsanity.com/paper/2503.18172/) (2025)
- [ChartCap: Mitigating Hallucination of Dense Chart Captioning](https://mlsanity.com/paper/2508.03164/) (2025)
- [Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions](https://mlsanity.com/paper/2409.19747/) (2024)
- [Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts](https://mlsanity.com/paper/2503.04095/) (2025)
- [ChartAdapter: Large Vision-Language Model for Chart Summarization](https://mlsanity.com/paper/2412.20715/) (2024)
- [Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023](https://mlsanity.com/paper/2501.19353/) (2025)
- [RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation](https://mlsanity.com/paper/2603.25804/) (2026)
- [POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering](https://mlsanity.com/paper/2507.11939/) (2025)
- [From Charts to Fair Narratives: Uncovering and Mitigating Geo-Economic Biases in Chart-to-Text](https://mlsanity.com/paper/2508.09450/) (2025)
- [ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models](https://mlsanity.com/paper/2605.23694/) (2026)
- [Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework](https://mlsanity.com/paper/2606.29808/) (2026)
- [Data Metabolism: An Efficient Data Design Schema For Vision Language Model](https://mlsanity.com/paper/2504.12316/) (2025)
- [Multi-Agent Multimodal Large Language Model Framework for Automated Interpretation of Fuel Efficiency Analytics in Public Transportation](https://mlsanity.com/paper/2511.13476/) (2025)
- [ChartAnchor: Chart Grounding with Structural-Semantic Fidelity](https://mlsanity.com/paper/2512.01017/) (2025)
- [Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement](https://mlsanity.com/paper/2602.10138/) (2026)
- [Beyond Description: A Multimodal Agent Framework for Insightful Chart Summarization](https://mlsanity.com/paper/2602.18731/) (2026)
- [DATAREEL: Automated Data-Driven Video Story Generation with Animations](https://mlsanity.com/paper/2604.25220/) (2026)
- [Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models](https://mlsanity.com/paper/2504.12315/) (2025)
- [Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks](https://mlsanity.com/paper/2510.06071/) (2025)
- [REChart: Reasoning-Efficient Chart Editing with Large Reasoning Models](https://mlsanity.com/paper/2608.17414/) (2026)

## Abstract (arXiv)

Automatic chart to text summarization is an effective tool for the visually impaired people along with providing precise insights of tabular data in natural language to the user. A large and well-structured dataset is always a key part for data driven models. In this paper, we propose ChartSumm: a large-scale benchmark dataset consisting of a total of 84,363 charts along with their metadata and descriptions covering a wide range of topics and chart types to generate short and long summaries. Extensive experiments with strong baseline models show that even though these models generate fluent and informative summaries by achieving decent scores in various automatic evaluation metrics, they often face issues like suffering from hallucination, missing out important data points, in addition to incorrect explanation of complex trends in the charts. We also investigated the potential of expanding ChartSumm to other languages using automated translation tools. These make our dataset a challenging benchmark for future research.