# Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Charts

arXiv: https://arxiv.org/abs/2101.08235 · HTML: https://mlsanity.com/paper/2101.08235/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jan 2021.

## Built on by (most cited first, 12 of 12)

- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021)
- [LineCap: Line Charts for Data Visualization Captioning Models](https://mlsanity.com/paper/2207.07243/) (2022)
- [VBridge: Connecting the Dots Between Features and Data to Explain Healthcare Models](https://mlsanity.com/paper/2108.02550/) (2021)
- [ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset](https://mlsanity.com/paper/2501.09349/) (2025)
- [C2: Scalable Auto-Feedback for LLM-based Chart Generation](https://mlsanity.com/paper/2410.18652/) (2024)
- [SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratings](https://mlsanity.com/paper/2403.17784/) (2024)
- [LaMP-Cap: Personalized Figure Caption Generation With Multimodal Figure Profiles](https://mlsanity.com/paper/2506.06561/) (2025)
- [ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models](https://mlsanity.com/paper/2605.23694/) (2026)
- [Visualization of Age Distributions as Elements of Medical Data-Stories](https://mlsanity.com/paper/2409.17854/) (2024)

## Abstract (arXiv)

Charts often contain visually prominent features that draw attention to aspects of the data and include text captions that emphasize aspects of the data. Through a crowdsourced study, we explore how readers gather takeaways when considering charts and captions together. We first ask participants to mark visually prominent regions in a set of line charts. We then generate text captions based on the prominent features and ask participants to report their takeaways after observing chart-caption pairs. We find that when both the chart and caption describe a high-prominence feature, readers treat the doubly emphasized high-prominence feature as the takeaway; when the caption describes a low-prominence chart feature, readers rely on the chart and report a higher-prominence feature as the takeaway. We also find that external information that provides context, helps further convey the caption's message to the reader. We use these findings to provide guidelines for authoring effective chart-caption pairs.