# ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset

arXiv: https://arxiv.org/abs/2501.09349 · HTML: https://mlsanity.com/paper/2501.09349/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 70 references that are in the map.
- First posted on arXiv 16 Jan 2025.

## Builds on (most cited first, 30 of 38)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration](https://mlsanity.com/paper/2307.05300/) (2023)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models](https://mlsanity.com/paper/2303.02927/) (2023)
- [ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning](https://mlsanity.com/paper/2401.02384/) (2024)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries](https://mlsanity.com/paper/2008.10723/) (2020)
- [ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language](https://mlsanity.com/paper/2311.01920/) (2023)
- [Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation](https://mlsanity.com/paper/2305.07375/) (2023)
- [From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models](https://mlsanity.com/paper/2403.12027/) (2024)
- [TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning](https://mlsanity.com/paper/2404.16635/) (2024)
- [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [LLM4Vis: Explainable Visualization Recommendation using ChatGPT](https://mlsanity.com/paper/2310.07652/) (2023)
- [DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articles](https://mlsanity.com/paper/2308.04076/) (2023)
- [LEVA: Using Large Language Models to Enhance Visual Analytics](https://mlsanity.com/paper/2403.05816/) (2024)
- [Generative AI for Visualization: State of the Art and Future Directions](https://mlsanity.com/paper/2404.18144/) (2024)
- [Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Models](https://mlsanity.com/paper/2309.10245/) (2023)

## Built on by (most cited first, 9 of 9)

- [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025)
- [Data-to-Dashboard: Multi-Agent LLM Framework for Insightful Visualization in Enterprise Analytics](https://mlsanity.com/paper/2505.23695/) (2025)
- [A2P-Vis: an Analyzer-to-Presenter Agentic Pipeline for Visual Insights Generation and Reporting](https://mlsanity.com/paper/2512.22101/) (2025)
- [ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models](https://mlsanity.com/paper/2605.23694/) (2026)
- [Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study](https://mlsanity.com/paper/2505.06702/) (2025)
- [Scaling Unverifiable Rewards: A Case Study on Visual Insights](https://mlsanity.com/paper/2512.22650/) (2025)
- [Beyond Description: A Multimodal Agent Framework for Insightful Chart Summarization](https://mlsanity.com/paper/2602.18731/) (2026)
- [DVBench: Benchmarking MLLMs for Understanding Dynamic Charts and Narratives in Data Videos](https://mlsanity.com/paper/2608.29711/) (2026)
- [WidgetVA: A Widget-Centric Framework and Benchmark for Agentic Visual Analytics](https://mlsanity.com/paper/2609.24094/) (2026)

## Abstract (arXiv)

Effective chart summary can significantly reduce the time and effort decision makers spend interpreting charts, enabling precise and efficient communication of data insights. Previous studies have faced challenges in generating accurate and semantically rich summaries of time-series data charts. In this paper, we identify summary elements and common hallucination types in the generation of time-series chart summaries, which serve as our guidelines for automatic generation. We introduce ChartInsighter, which automatically generates chart summaries of time-series data, effectively reducing hallucinations in chart summary generation. Specifically, we assign multiple agents to generate the initial chart summary and collaborate iteratively, during which they invoke external data analysis modules to extract insights and compile them into a coherent summary. Additionally, we implement a self-consistency test method to validate and correct our summary. We create a high-quality benchmark of charts and summaries, with hallucination types annotated on a sentence-by-sentence basis, facilitating the evaluation of the effectiveness of reducing hallucinations. Our evaluations using our benchmark show that our method surpasses state-of-the-art models, and that our summary hallucination rate is the lowest, which effectively reduces various hallucinations and improves summary quality. The benchmark is available at https://github.com/wangfen01/ChartInsighter.