# InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions

arXiv: https://arxiv.org/abs/2503.04110 · HTML: https://mlsanity.com/paper/2503.04110/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Builds on (most cited first, 28 of 28)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models](https://mlsanity.com/paper/2303.02927/) (2023)
- [NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries](https://mlsanity.com/paper/2008.10723/) (2020)
- [ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language](https://mlsanity.com/paper/2311.01920/) (2023)
- [Towards Natural Language Interfaces for Data Visualization: A Survey](https://mlsanity.com/paper/2109.03506/) (2021)
- [AI4VIS: Survey on Artificial Intelligence Approaches for Data Visualization](https://mlsanity.com/paper/2102.01330/) (2021)
- [DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models](https://mlsanity.com/paper/2310.03691/) (2023)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning](https://mlsanity.com/paper/2407.20174/) (2024)
- [LLM4Vis: Explainable Visualization Recommendation using ChatGPT](https://mlsanity.com/paper/2310.07652/) (2023)
- [DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articles](https://mlsanity.com/paper/2308.04076/) (2023)
- [LEVA: Using Large Language Models to Enhance Visual Analytics](https://mlsanity.com/paper/2403.05816/) (2024)
- [Generative AI for Visualization: State of the Art and Future Directions](https://mlsanity.com/paper/2404.18144/) (2024)
- [Towards Natural Language-Based Visualization Authoring](https://mlsanity.com/paper/2208.10947/) (2022)
- [Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model](https://mlsanity.com/paper/2304.14630/) (2023)
- [Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Models](https://mlsanity.com/paper/2309.10245/) (2023)
- [Are LLMs ready for Visualization?](https://mlsanity.com/paper/2403.06158/) (2024)
- [From Data to Story: Towards Automatic Animated Data Video Creation with LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2408.03876/) (2024)
- [Prompt4Vis: Prompting Large Language Models with Example Mining and Schema Filtering for Tabular Data Visualization](https://mlsanity.com/paper/2402.07909/) (2024)
- [Data Player: Automatic Generation of Data Videos with Narration-Animation Interplay](https://mlsanity.com/paper/2308.04703/) (2023)
- [Reviving Static Charts into Live Charts](https://mlsanity.com/paper/2309.02967/) (2023)
- [Interweaving Multimodal Interaction with Flexible Unit Visualizations for Data Exploration](https://mlsanity.com/paper/2004.10428/) (2020)
- [ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models](https://mlsanity.com/paper/2306.09649/) (2023)
- [SlopeSeeker: A Search Tool for Exploring a Dataset of Quantifiable Trends](https://mlsanity.com/paper/2402.12214/) (2024)

## Built on by (most cited first, 13 of 13)

- [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025)
- [Exploring Multimodal Prompt for Visualization Authoring with Large Language Models](https://mlsanity.com/paper/2504.13700/) (2025)
- [Plover: Steering GUI Agents through Plan-Centric Interaction](https://mlsanity.com/paper/2607.15193/) (2026)
- [Multimedia and Visual Analytics in the Agentic Era](https://mlsanity.com/paper/2504.06138/) (2025)
- [A Generative AI System for Biomedical Data Discovery with Grammar-Based Visualizations](https://mlsanity.com/paper/2509.16454/) (2025)
- [YAC: Bridging Natural Language and Interactive Visual Exploration with Generative AI for Biomedical Data Discovery](https://mlsanity.com/paper/2509.19182/) (2025)
- [AwesomeLit: Towards Hypothesis Generation with Agent-Supported Literature Research](https://mlsanity.com/paper/2603.22648/) (2026)
- [VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents](https://mlsanity.com/paper/2506.21582/) (2025)
- [Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation](https://mlsanity.com/paper/2604.14668/) (2026)
- [Agentic Authoring of Interactive Multiview Visualizations in Genomics](https://mlsanity.com/paper/2606.00370/) (2026)
- [Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy](https://mlsanity.com/paper/2606.03142/) (2026)
- [AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysis](https://mlsanity.com/paper/2608.17834/) (2026)
- [Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses](https://mlsanity.com/paper/2609.35255/) (2026)

## Abstract (arXiv)

The rise of Large Language Models (LLMs) and generative visual analytics systems has transformed data-driven insights, yet significant challenges persist in accurately interpreting users' analytical and interaction intents. While language inputs offer flexibility, they often lack precision, making the expression of complex intents inefficient, error-prone, and time-intensive. To address these limitations, we investigate the design space of multimodal interactions for generative visual analytics through a literature review and pilot brainstorming sessions. Building on these insights, we introduce a highly extensible workflow that integrates multiple LLM agents for intent inference and visualization generation. We develop InterChat, a generative visual analytics system that combines direct manipulation of visual elements with natural language inputs. This integration enables precise intent communication and supports progressive, visually driven exploratory data analyses. By employing effective prompt engineering, and contextual interaction linking, alongside intuitive visualization and interaction designs, InterChat bridges the gap between user interactions and LLM-driven visualizations, enhancing both interpretability and usability. Extensive evaluations, including two usage scenarios, a user study, and expert feedback, demonstrate the effectiveness of InterChat. Results show significant improvements in the accuracy and efficiency of handling complex visual analytics tasks, highlighting the potential of multimodal interactions to redefine user engagement and analytical depth in generative visual analytics.