# How Aligned are Human Chart Takeaways and LLM Predictions? A Case Study on Bar Charts with Varying Layouts

arXiv: https://arxiv.org/abs/2408.06837 · HTML: https://mlsanity.com/paper/2408.06837/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2024.

## Built on by (most cited first, 10 of 10)

- [Benchmarking Visual Language Models on Standardized Visualization Literacy Tests](https://mlsanity.com/paper/2503.16632/) (2025)
- [Data-to-Dashboard: Multi-Agent LLM Framework for Insightful Visualization in Enterprise Analytics](https://mlsanity.com/paper/2505.23695/) (2025)
- [MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces](https://mlsanity.com/paper/2510.08783/) (2025)
- [See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs](https://mlsanity.com/paper/2504.09809/) (2025)
- [Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study](https://mlsanity.com/paper/2505.06702/) (2025)
- [A Rigorous Behavior Assessment of CNNs Using a Data-Domain Sampling Regime](https://mlsanity.com/paper/2507.03866/) (2025)
- [Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks](https://mlsanity.com/paper/2510.06071/) (2025)
- [GPT-5 Model Corrected GPT-4V's Chart Reading Errors, Not Prompting](https://mlsanity.com/paper/2510.06782/) (2025)
- [Beauty in the Eye of AI: Aligning LLMs and Vision Models with Human Aesthetics in Network Visualization](https://mlsanity.com/paper/2604.03417/) (2026)
- [EvidenceLens: A Claim-Evidence Matrix for Auditing Financial Question Answering](https://mlsanity.com/paper/2606.23724/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have been adopted for a variety of visualizations tasks, but how far are we from perceptually aware LLMs that can predict human takeaways? Graphical perception literature has shown that human chart takeaways are sensitive to visualization design choices, such as spatial layouts. In this work, we examine the extent to which LLMs exhibit such sensitivity when generating takeaways, using bar charts with varying spatial layouts as a case study. We conducted three experiments and tested four common bar chart layouts: vertically juxtaposed, horizontally juxtaposed, overlaid, and stacked. In Experiment 1, we identified the optimal configurations to generate meaningful chart takeaways by testing four LLMs, two temperature settings, nine chart specifications, and two prompting strategies. We found that even state-of-the-art LLMs struggled to generate semantically diverse and factually accurate takeaways. In Experiment 2, we used the optimal configurations to generate 30 chart takeaways each for eight visualizations across four layouts and two datasets in both zero-shot and one-shot settings. Compared to human takeaways, we found that the takeaways LLMs generated often did not match the types of comparisons made by humans. In Experiment 3, we examined the effect of chart context and data on LLM takeaways. We found that LLMs, unlike humans, exhibited variation in takeaway comparison types for different bar charts using the same bar layout. Overall, our case study evaluates the ability of LLMs to emulate human interpretations of data and points to challenges and opportunities in using LLMs to predict human chart takeaways.