# Vi(E)va LLM! A Conceptual Stack for Evaluating and Interpreting Generative AI-based Visualizations

arXiv: https://arxiv.org/abs/2402.02167 · HTML: https://mlsanity.com/paper/2402.02167/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 67 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Builds on (most cited first, 30 of 35)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning](https://mlsanity.com/paper/1711.04436/) (2017)
- [NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries](https://mlsanity.com/paper/2008.10723/) (2020)
- [ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language](https://mlsanity.com/paper/2311.01920/) (2023)
- [Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models](https://mlsanity.com/paper/2302.02094/) (2023)
- [E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles](https://mlsanity.com/paper/1906.03973/) (2019)
- [Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data](https://mlsanity.com/paper/2106.05006/) (2021)
- [Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations](https://mlsanity.com/paper/2110.00680/) (2021)
- [nvBench: A Large-Scale Synthesized Dataset for Cross-Domain Natural Language to Visualization Task](https://mlsanity.com/paper/2112.12926/) (2021)
- [How Good is ChatGPT in Giving Advice on Your Visualization Design?](https://mlsanity.com/paper/2310.09617/) (2023)
- [A Survey of Binary Code Similarity](https://mlsanity.com/paper/1909.11424/) (2019)
- [Beyond Generating Code: Evaluating GPT on a Data Visualization Course](https://mlsanity.com/paper/2306.02914/) (2023)
- [PI2: End-to-end Interactive Visualization Interface Generation from Queries](https://mlsanity.com/paper/2107.08203/) (2021)
- [Image similarity using Deep CNN and Curriculum Learning](https://mlsanity.com/paper/1709.08761/) (2017)
- [Quda: Natural Language Queries for Visual Data Analytics](https://mlsanity.com/paper/2005.03257/) (2020)

## Built on by (most cited first, 10 of 10)

- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generation](https://mlsanity.com/paper/2506.03930/) (2025)
- [AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy](https://mlsanity.com/paper/2505.20538/) (2025)
- [Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction](https://mlsanity.com/paper/2508.04842/) (2025)
- [From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualization](https://mlsanity.com/paper/2505.16832/) (2025)
- [V-RECS, a Low-Cost LLM4VIS Recommender with Explanations, Captioning and Suggestions](https://mlsanity.com/paper/2406.15259/) (2024)
- [Multi-Agent Actor-Critic Generative AI for Query Resolution and Analysis](https://mlsanity.com/paper/2502.13164/) (2025)
- [Charting the Future: Using Chart Question-Answering for Scalable Evaluation of LLM-Driven Data Visualizations](https://mlsanity.com/paper/2409.18764/) (2024)
- [Does It Run and Is That Enough? Revisiting Text-to-Chart Generation with a Multi-Agent Approach](https://mlsanity.com/paper/2506.06175/) (2025)
- [Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents](https://mlsanity.com/paper/2609.17842/) (2026)

## Abstract (arXiv)

The automatic generation of visualizations is an old task that, through the years, has shown more and more interest from the research and practitioner communities. Recently, large language models (LLM) have become an interesting option for supporting generative tasks related to visualization, demonstrating initial promising results. At the same time, several pitfalls, like the multiple ways of instructing an LLM to generate the desired result, the different perspectives leading the generation (code-based, image-based, grammar-based), and the presence of hallucinations even for the visualization generation task, make their usage less affordable than expected. Following similar initiatives for benchmarking LLMs, this paper copes with the problem of modeling the evaluation of a generated visualization through an LLM. We propose a theoretical evaluation stack, EvaLLM, that decomposes the evaluation effort in its atomic components, characterizes their nature, and provides an overview of how to implement and interpret them. We also designed and implemented an evaluation platform that provides a benchmarking resource for the visualization generation task. The platform supports automatic and manual scoring conducted by multiple assessors to support a fine-grained and semantic evaluation based on the EvaLLM stack. Two case studies on GPT3.5-turbo with Code Interpreter and Llama2-70-b models show the benefits of EvaLLM and illustrate interesting results on the current state-of-the-art LLM-generated visualizations.