# Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study

arXiv: https://arxiv.org/abs/2505.06702 · HTML: https://mlsanity.com/paper/2505.06702/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 10 May 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)
- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 22 of 22)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [AgentTuning: Enabling Generalized Agent Abilities for LLMs](https://mlsanity.com/paper/2310.12823/) (2023)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [Visualization Generation with Large Language Models: An Evaluation](https://mlsanity.com/paper/2401.11255/) (2024)
- [Revisiting Dimensionality Reduction Techniques for Visual Cluster Analysis: An Empirical Study](https://mlsanity.com/paper/2110.02894/) (2021)
- [How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?](https://mlsanity.com/paper/2407.17291/) (2024)
- [LEVA: Using Large Language Models to Enhance Visual Analytics](https://mlsanity.com/paper/2403.05816/) (2024)
- [Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Models](https://mlsanity.com/paper/2309.10245/) (2023)
- [Exploring the Capability of LLMs in Performing Low-Level Visual Analytic Tasks on SVG Data Visualizations](https://mlsanity.com/paper/2404.19097/) (2024)
- [Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows](https://mlsanity.com/paper/2312.11681/) (2023)
- [Data Player: Automatic Generation of Data Videos with Narration-Animation Interplay](https://mlsanity.com/paper/2308.04703/) (2023)
- [Fine-Tuned Large Language Model for Visualization System: A Study on Self-Regulated Learning in Education](https://mlsanity.com/paper/2407.20570/) (2024)
- [DracoGPT: Extracting Visualization Design Preferences from Large Language Models](https://mlsanity.com/paper/2408.06845/) (2024)
- [How Aligned are Human Chart Takeaways and LLM Predictions? A Case Study on Bar Charts with Varying Layouts](https://mlsanity.com/paper/2408.06837/) (2024)
- [Visualization Literacy of Multimodal Large Language Models: A Comparative Study](https://mlsanity.com/paper/2407.10996/) (2024)
- [Narrative Player: Reviving Data Narratives with Visuals](https://mlsanity.com/paper/2410.03268/) (2024)
- [ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset](https://mlsanity.com/paper/2501.09349/) (2025)
- [Generalization of CNNs on Relational Reasoning with Bar Charts](https://mlsanity.com/paper/2503.00086/) (2025)
- [Evaluating the Effect of Timeline Shape on Visualization Task Performance](https://mlsanity.com/paper/2005.06039/) (2020)
- [Design Characterization for Black-and-White Textures in Visualization](https://mlsanity.com/paper/2307.10089/) (2023)

## Built on by (most cited first, 1 of 1)

- [Can Agents Judge Systematic Reviews Like Humans? Evaluating SLRs with LLM-based Multi-Agent System](https://mlsanity.com/paper/2509.17240/) (2025)

## Abstract (arXiv)

Large language models encode knowledge in various domains and demonstrate the ability to understand visualizations. They may also capture visualization design knowledge and potentially help reduce the cost of formative studies. However, it remains a question whether large language models are capable of predicting human feedback on visualizations. To investigate this question, we conducted three studies to examine whether large model-based agents can simulate human ratings in visualization tasks. The first study, replicating a published study involving human subjects, shows agents are promising in conducting human-like reasoning and rating, and its result guides the subsequent experimental design. The second study repeated six human-subject studies reported in literature on subjective ratings, but replacing human participants with agents. Consulting with five human experts, this study demonstrates that the alignment of agent ratings with human ratings positively correlates with the confidence levels of the experts before the experiments. The third study tests commonly used techniques for enhancing agents, including preprocessing visual and textual inputs, and knowledge injection. The results reveal the issues of these techniques in robustness and potential induction of biases. The three studies indicate that language model-based agents can potentially simulate human ratings in visualization experiments, provided that they are guided by high-confidence hypotheses from expert evaluators. Additionally, we demonstrate the usage scenario of swiftly evaluating prototypes with agents. We discuss insights and future directions for evaluating and improving the alignment of agent ratings with human ratings. We note that simulation may only serve as complements and cannot replace user studies.