# Beyond Static Charts: Can Language and Vision Language Models Generate Interactive Data Visualization Interfaces?

arXiv: https://arxiv.org/abs/2609.26208 · HTML: https://mlsanity.com/paper/2609.26208/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 14 of 14)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing](https://mlsanity.com/paper/2401.10880/) (2024)
- [Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations](https://mlsanity.com/paper/2110.00680/) (2021)
- [Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping](https://mlsanity.com/paper/2410.16232/) (2024)
- [nvBench: A Large-Scale Synthesized Dataset for Cross-Domain Natural Language to Visualization Task](https://mlsanity.com/paper/2112.12926/) (2021)
- [Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model](https://mlsanity.com/paper/2503.04084/) (2025)
- [UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs](https://mlsanity.com/paper/2505.09904/) (2025)
- [PI2: End-to-end Interactive Visualization Interface Generation from Queries](https://mlsanity.com/paper/2107.08203/) (2021)
- [Generative UI: LLMs are Effective UI Generators](https://mlsanity.com/paper/2604.09577/) (2026)
- [Challenges and Opportunities in Data Visualization Education: A Call to Action](https://mlsanity.com/paper/2308.07703/) (2023)
- [Aligning Text, Code, and Vision: A Multi-Objective Reinforcement Learning Framework for Text-to-Visualization](https://mlsanity.com/paper/2601.04582/) (2026)

## Abstract (arXiv)

Data visualization is central to analytical reasoning, but real-world analysis increasingly requires language-driven interactive interfaces rather than static charts. Although recent large language and vision language models (LLMs/VLMs) have shown promise in generating static charts from natural language, their ability to generate interactive data visualization interfaces remains largely unexplored due to the lack of benchmarks. We introduce VIS-GEN, a benchmark for evaluating how well LLMs/VLMs can generate interactive visualization interfaces from natural language queries. VIS-GEN comprises 3,042 samples covering diverse analytical intents, including data filtering, temporal analysis, and visualization editing, each paired with dataset metadata and natural language queries that are designed to reflect realistic, goal driven data exploration scenarios. We benchmark 14 state-of-the-art open-source and closed-source LLMs/VLMs, revealing large performance gaps and frequent failures on queries involving implicit intent, multiple interaction alternatives, and complex editing operations, highlighting interactive interface generation as a key open challenge beyond static chart synthesis. To address this, we propose a structured multi stage interface generation framework that decomposes the task into visualization design representation, generation of multiple interface candidates, constraint-aware critique, and self-refinement. This approach improves the best models pass rate by 15.9 percentage points, demonstrating a practical path toward more reliable language-driven interactive visualization systems. We release VIS-GEN at https://github.com/vis-nlp/VIS-GEN.