# LLM-Assisted Visual Analytics: Opportunities and Challenges

arXiv: https://arxiv.org/abs/2409.02691 · HTML: https://mlsanity.com/paper/2409.02691/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 60 references that are in the map.
- First posted on arXiv 4 Sep 2024.

## Builds on (most cited first, 28 of 28)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models](https://mlsanity.com/paper/2303.02927/) (2023)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow](https://mlsanity.com/paper/2306.07209/) (2023)
- [NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries](https://mlsanity.com/paper/2008.10723/) (2020)
- [ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language](https://mlsanity.com/paper/2311.01920/) (2023)
- [Towards Natural Language Interfaces for Data Visualization: A Survey](https://mlsanity.com/paper/2109.03506/) (2021)
- [Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models](https://mlsanity.com/paper/2302.02094/) (2023)
- [AI4VIS: Survey on Artificial Intelligence Approaches for Data Visualization](https://mlsanity.com/paper/2102.01330/) (2021)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [Large Language Models are Efficient Learners of Noise-Robust Speech Recognition](https://mlsanity.com/paper/2401.10446/) (2024)
- [Calliope: Automatic Visual Data Story Generation from a Spreadsheet](https://mlsanity.com/paper/2010.09975/) (2020)
- [A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization](https://mlsanity.com/paper/2012.00467/) (2020)
- [How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?](https://mlsanity.com/paper/2407.17291/) (2024)
- [Chart Question Answering: State of the Art and Future Directions](https://mlsanity.com/paper/2205.03966/) (2022)
- [DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articles](https://mlsanity.com/paper/2308.04076/) (2023)
- [LEVA: Using Large Language Models to Enhance Visual Analytics](https://mlsanity.com/paper/2403.05816/) (2024)
- [Generative AI for Visualization: State of the Art and Future Directions](https://mlsanity.com/paper/2404.18144/) (2024)
- [Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model](https://mlsanity.com/paper/2304.14630/) (2023)
- [Facilitating Conversational Interaction in Natural Language Interfaces for Visualization](https://mlsanity.com/paper/2207.00189/) (2022)
- [InkSight: Leveraging Sketch Interaction for Documenting Chart Findings in Computational Notebooks](https://mlsanity.com/paper/2307.07922/) (2023)
- [PhenoFlow: A Human-LLM Driven Visual Analytics System for Exploring Large and Complex Stroke Datasets](https://mlsanity.com/paper/2407.16329/) (2024)
- [VizCommender: Computing Text-Based Similarity in Visualization Repositories for Content-Based Recommendations](https://mlsanity.com/paper/2008.07702/) (2020)
- [Probing Language Models' Gesture Understanding for Enhanced Human-AI Interaction](https://mlsanity.com/paper/2401.17858/) (2024)

## Built on by (most cited first, 2 of 2)

- [CoDA: Agentic Systems for Collaborative Data Visualization](https://mlsanity.com/paper/2510.03194/) (2025)
- [ClimateSOM: A Visual Analysis Workflow for Climate Ensemble Datasets](https://mlsanity.com/paper/2508.06732/) (2025)

## Abstract (arXiv)

We explore the integration of large language models (LLMs) into visual analytics (VA) systems to transform their capabilities through intuitive natural language interactions. We survey current research directions in this emerging field, examining how LLMs are integrated into data management, language interaction, visualisation generation, and language generation processes. We highlight the new possibilities that LLMs bring to VA, especially how they can change VA processes beyond the usual use cases. We especially highlight building new visualisation-language models, allowing access of a breadth of domain knowledge, multimodal interaction, and opportunities with guidance. Finally, we carefully consider the prominent challenges of using current LLMs in VA tasks. Our discussions in this paper aim to guide future researchers working on LLM-assisted VA systems and help them navigate common obstacles when developing these systems.