# Tackling Hallucinations in Neural Chart Summarization

arXiv: https://arxiv.org/abs/2308.00399 · HTML: https://mlsanity.com/paper/2308.00399/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey](https://mlsanity.com/paper/2104.14839/) (2021)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference](https://mlsanity.com/paper/2011.10819/) (2020)
- [Controlling Hallucinations at Word Level in Data-to-Text Generation](https://mlsanity.com/paper/2102.02810/) (2021)
- [NUBIA: NeUral Based Interchangeability Assessor for Text Generation](https://mlsanity.com/paper/2004.14667/) (2020)
- [A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems](https://mlsanity.com/paper/2011.03992/) (2020)
- [Revisiting Challenges in Data-to-Text Generation with Fact Grounding](https://mlsanity.com/paper/2001.03830/) (2020)
- [AutoChart: A Dataset for Chart-to-Text Generation Task](https://mlsanity.com/paper/2108.06897/) (2021)
- [AgreeSum: Agreement-Oriented Multi-Document Summarization](https://mlsanity.com/paper/2106.02278/) (2021)
- [MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization](https://mlsanity.com/paper/2109.10650/) (2021)

## Built on by (most cited first, 6 of 6)

- [An Audit on the Perspectives and Challenges of Hallucinations in NLP](https://mlsanity.com/paper/2404.07461/) (2024)
- [ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset](https://mlsanity.com/paper/2501.09349/) (2025)
- [Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions](https://mlsanity.com/paper/2409.19747/) (2024)
- [How Much Do LLMs Hallucinate across Languages? On Realistic Multilingual Estimation of LLM Hallucination](https://mlsanity.com/paper/2502.12769/) (2025)
- [An Extensive Evaluation of Factual Consistency in Large Language Models for Data-to-Text Generation](https://mlsanity.com/paper/2411.19203/) (2024)
- [Impact of Model Size on Fine-tuned LLM Performance in Data-to-Text Generation: A State-of-the-Art Investigation](https://mlsanity.com/paper/2407.14088/) (2024)

## Abstract (arXiv)

Hallucinations in text generation occur when the system produces text that is not grounded in the input. In this work, we tackle the problem of hallucinations in neural chart summarization. Our analysis shows that the target side of chart summarization training datasets often contains additional information, leading to hallucinations. We propose a natural language inference (NLI) based method to preprocess the training data and show through human evaluation that our method significantly reduces hallucinations. We also found that shortening long-distance dependencies in the input sequence and adding chart-related information like title and legends improves the overall performance.