# Where LLMs Fail with Visualization DSLs

arXiv: https://arxiv.org/abs/2610.01873 · HTML: https://mlsanity.com/paper/2610.01873/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Builds on (most cited first, 21 of 21)

- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation](https://mlsanity.com/paper/2401.06591/) (2024)
- [StarVector: Generating Scalable Vector Graphics Code from Images and Text](https://mlsanity.com/paper/2312.11556/) (2023)
- [LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models](https://mlsanity.com/paper/2303.02927/) (2023)
- [NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries](https://mlsanity.com/paper/2008.10723/) (2020)
- [Grammar Prompting for Domain-Specific Language Generation with Large Language Models](https://mlsanity.com/paper/2305.19234/) (2023)
- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [Data Formulator: AI-powered Concept-driven Visualization Authoring](https://mlsanity.com/paper/2309.10094/) (2023)
- [Visualization Generation with Large Language Models: An Evaluation](https://mlsanity.com/paper/2401.11255/) (2024)
- [Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning](https://mlsanity.com/paper/2407.20174/) (2024)
- [Data Formulator 2: Iterative Creation of Data Visualizations, with AI Transforming Data Along the Way](https://mlsanity.com/paper/2408.16119/) (2024)
- [ChartMark: A Structured Grammar for Chart Annotation](https://mlsanity.com/paper/2507.21810/) (2025)
- [Integrated Visualization Editing via Parameterized Declarative Templates](https://mlsanity.com/paper/2101.07902/) (2021)
- [Critical Reflections on Visualization Authoring Systems](https://mlsanity.com/paper/1907.13568/) (2019)
- [No Grammar to Rule Them All: A Survey of JSON-style DSLs for Visualization](https://mlsanity.com/paper/2207.07998/) (2022)
- [A Survey on Annotations in Information Visualization: Empirical Insights, Applications, and Challenges](https://mlsanity.com/paper/2410.05579/) (2024)
- [Lyra 2: Designing Interactive Visualizations by Demonstration](https://mlsanity.com/paper/2008.09576/) (2020)
- [Encodable: Configurable Grammar for Visualization Components](https://mlsanity.com/paper/2009.00722/) (2020)
- [Gemini: A Grammar and Recommender System for AnimatedTransitions in Statistical Graphics](https://mlsanity.com/paper/2009.01429/) (2020)
- [Metrics-Based Evaluation and Comparison of Visualization Notations](https://mlsanity.com/paper/2308.16353/) (2023)
- [Flint: A Semantics-Driven Data Visualization Intermediate Language](https://mlsanity.com/paper/2607.20775/) (2026)

## Abstract (arXiv)

As LLMs take up the role of authoring charts using visualization domain-specific languages (DSLs), the human constraints that shaped those languages may no longer apply, as what is easy for a person is not necessarily easy for a model. To understand how LLMs might work better with DSLs, we explore where and how they fail with current DSL designs. We evaluate 10 JSON-style visualization DSLs with 41 tasks across 3 LLMs, then assess the generated specifications with JSON and rendering checks, and qualitative coding of failed cases. Analyzing how this specification generation process fails, we identify four recurring failure patterns, link each to specific DSL features, and discuss design considerations for future DSL designs.