# Towards Natural Language-Based Visualization Authoring

arXiv: https://arxiv.org/abs/2208.10947 · HTML: https://mlsanity.com/paper/2208.10947/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2022.

## Built on by (most cited first, 22 of 22)

- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models](https://mlsanity.com/paper/2303.02927/) (2023)
- [ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering](https://mlsanity.com/paper/2405.07001/) (2024)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs](https://mlsanity.com/paper/2307.10168/) (2023)
- [DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articles](https://mlsanity.com/paper/2308.04076/) (2023)
- [Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model](https://mlsanity.com/paper/2304.14630/) (2023)
- [XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis](https://mlsanity.com/paper/2301.10385/) (2023)
- [From Data to Story: Towards Automatic Animated Data Video Creation with LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2408.03876/) (2024)
- [InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions](https://mlsanity.com/paper/2503.04110/) (2025)
- [ParaView-MCP: An Autonomous Visualization Agent with Direct Tool Use](https://mlsanity.com/paper/2505.07064/) (2025)
- [TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generation](https://mlsanity.com/paper/2401.11094/) (2024)
- [DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning](https://mlsanity.com/paper/2508.01700/) (2025)
- [Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytelling](https://mlsanity.com/paper/2304.08366/) (2023)
- [Exploring Multimodal Prompt for Visualization Authoring with Large Language Models](https://mlsanity.com/paper/2504.13700/) (2025)
- [MultiVis-Agent: A Multi-Agent Framework with Logic Rules for Reliable and Comprehensive Cross-Modal Data Visualization](https://mlsanity.com/paper/2601.18320/) (2026)
- [InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs](https://mlsanity.com/paper/2312.09672/) (2023)
- [A Generative AI System for Biomedical Data Discovery with Grammar-Based Visualizations](https://mlsanity.com/paper/2509.16454/) (2025)
- [YAC: Bridging Natural Language and Interactive Visual Exploration with Generative AI for Biomedical Data Discovery](https://mlsanity.com/paper/2509.19182/) (2025)
- [NOVAID: Natural-language Observability Visualization Assistant for ITOps Dashboard Widget Generation](https://mlsanity.com/paper/2601.11531/) (2025)
- [From Static to Interactive: Authoring Interactive Visualizations via Natural Language](https://mlsanity.com/paper/2601.17736/) (2026)
- [Show Me the Infographic I Imagine: Intent-Aware Infographic Retrieval for Authoring Support](https://mlsanity.com/paper/2604.07989/) (2026)

## Abstract (arXiv)

A key challenge to visualization authoring is the process of getting familiar with the complex user interfaces of authoring tools. Natural Language Interface (NLI) presents promising benefits due to its learnability and usability. However, supporting NLIs for authoring tools requires expertise in natural language processing, while existing NLIs are mostly designed for visual analytic workflow. In this paper, we propose an authoring-oriented NLI pipeline by introducing a structured representation of users' visualization editing intents, called editing actions, based on a formative study and an extensive survey on visualization construction tools. The editing actions are executable, and thus decouple natural language interpretation and visualization applications as an intermediate layer. We implement a deep learning-based NL interpreter to translate NL utterances into editing actions. The interpreter is reusable and extensible across authoring tools. The authoring tools only need to map the editing actions into tool-specific operations. To illustrate the usages of the NL interpreter, we implement an Excel chart editor and a proof-of-concept authoring tool, VisTalk. We conduct a user study with VisTalk to understand the usage patterns of NL-based authoring systems. Finally, we discuss observations on how users author charts with natural language, as well as implications for future research.