# Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Models

arXiv: https://arxiv.org/abs/2309.10245 · HTML: https://mlsanity.com/paper/2309.10245/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2023.

## Built on by (most cited first, 20 of 20)

- [ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning](https://mlsanity.com/paper/2401.02384/) (2024)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning](https://mlsanity.com/paper/2407.20174/) (2024)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions](https://mlsanity.com/paper/2503.04110/) (2025)
- [ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset](https://mlsanity.com/paper/2501.09349/) (2025)
- [Exploring Multimodal Prompt for Visualization Authoring with Large Language Models](https://mlsanity.com/paper/2504.13700/) (2025)
- [Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap](https://mlsanity.com/paper/2512.04680/) (2025)
- [V-RECS, a Low-Cost LLM4VIS Recommender with Explanations, Captioning and Suggestions](https://mlsanity.com/paper/2406.15259/) (2024)
- [Towards Understanding Graphical Perception in Large Multimodal Models](https://mlsanity.com/paper/2503.10857/) (2025)
- [MultiVis-Agent: A Multi-Agent Framework with Logic Rules for Reliable and Comprehensive Cross-Modal Data Visualization](https://mlsanity.com/paper/2601.18320/) (2026)
- [ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models](https://mlsanity.com/paper/2605.23694/) (2026)
- [A Review on Large Language Models for Visual Analytics](https://mlsanity.com/paper/2503.15176/) (2025)
- [Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study](https://mlsanity.com/paper/2505.06702/) (2025)
- [Beyond Description: A Multimodal Agent Framework for Insightful Chart Summarization](https://mlsanity.com/paper/2602.18731/) (2026)
- [GQVis: A Dataset of Genomics Data Questions and Visualizations for Generative AI](https://mlsanity.com/paper/2510.13816/) (2025)
- [Toward a Machine Bertin: Why Visualization Needs Design Principles for Machine Cognition](https://mlsanity.com/paper/2602.01527/) (2026)
- [Agentic Authoring of Interactive Multiview Visualizations in Genomics](https://mlsanity.com/paper/2606.00370/) (2026)
- [Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy](https://mlsanity.com/paper/2606.03142/) (2026)
- [VisInteract: Towards Dynamic Interactive Text-to-Visualization under Imperfect Queries](https://mlsanity.com/paper/2609.15182/) (2026)

## Abstract (arXiv)

We introduce VL2NL, a Large Language Model (LLM) framework that generates rich and diverse NL datasets using only Vega-Lite specifications as input, thereby streamlining the development of Natural Language Interfaces (NLIs) for data visualization. To synthesize relevant chart semantics accurately and enhance syntactic diversity in each NL dataset, we leverage 1) a guided discovery incorporated into prompting so that LLMs can steer themselves to create faithful NL datasets in a self-directed manner; 2) a score-based paraphrasing to augment NL syntax along with four language axes. We also present a new collection of 1,981 real-world Vega-Lite specifications that have increased diversity and complexity than existing chart collections. When tested on our chart collection, VL2NL extracted chart semantics and generated L1/L2 captions with 89.4% and 76.0% accuracy, respectively. It also demonstrated generating and paraphrasing utterances and questions with greater diversity compared to the benchmarks. Last, we discuss how our NL datasets and framework can be utilized in real-world scenarios. The codes and chart collection are available at https://github.com/hyungkwonko/chart-llm.