# LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models

arXiv: https://arxiv.org/abs/2303.02927 · HTML: https://mlsanity.com/paper/2303.02927/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries](https://mlsanity.com/paper/2008.10723/) (2020)
- [Data2Vis: Automatic Generation of Data Visualizations Using Sequence to Sequence Recurrent Neural Networks](https://mlsanity.com/paper/1804.03126/) (2018)
- [Towards Natural Language-Based Visualization Authoring](https://mlsanity.com/paper/2208.10947/) (2022)
- [Facilitating Conversational Interaction in Natural Language Interfaces for Visualization](https://mlsanity.com/paper/2207.00189/) (2022)
- [Type-Directed Synthesis of Visualizations from Natural Language Queries](https://mlsanity.com/paper/2209.01081/) (2022)
- [Agnostic Visual Recommendation Systems: Open Challenges and Future Directions](https://mlsanity.com/paper/2302.00569/) (2023)
- [Sato: Contextual Semantic Type Detection in Tables](https://mlsanity.com/paper/1911.06311/) (2019)

## Built on by (most cited first, 30 of 82)

- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [BLADE: Benchmarking Language Model Agents for Data-Driven Science](https://mlsanity.com/paper/2408.09667/) (2024)
- [Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow](https://mlsanity.com/paper/2306.07209/) (2023)
- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [Data Formulator: AI-powered Concept-driven Visualization Authoring](https://mlsanity.com/paper/2309.10094/) (2023)
- [PaperBanana: Automating Academic Illustration for AI Scientists](https://mlsanity.com/paper/2601.23265/) (2026)
- [Generative AI for Visualization: State of the Art and Future Directions](https://mlsanity.com/paper/2404.18144/) (2024)
- [AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Making](https://mlsanity.com/paper/2312.04494/) (2023)
- [LightVA: Lightweight Visual Analytics with LLM Agent-Based Task Planning and Execution](https://mlsanity.com/paper/2411.05651/) (2024)
- [From Data to Story: Towards Automatic Animated Data Video Creation with LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2408.03876/) (2024)
- [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024)
- [ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent](https://mlsanity.com/paper/2507.18165/) (2025)
- [Where Are We So Far? Understanding Data Storytelling Tools from the Perspective of Human-AI Collaboration](https://mlsanity.com/paper/2309.15723/) (2023)
- [InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions](https://mlsanity.com/paper/2503.04110/) (2025)
- [AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy](https://mlsanity.com/paper/2505.20538/) (2025)
- [BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks](https://mlsanity.com/paper/2404.07387/) (2024)
- [SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents](https://mlsanity.com/paper/2603.29139/) (2026)
- [A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future](https://mlsanity.com/paper/2412.14056/) (2024)
- [ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset](https://mlsanity.com/paper/2501.09349/) (2025)
- [VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation](https://mlsanity.com/paper/2506.13326/) (2025)
- [DataLab: A Unified Platform for LLM-Powered Business Intelligence](https://mlsanity.com/paper/2412.02205/) (2024)
- [SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science](https://mlsanity.com/paper/2503.23314/) (2025)
- [Just-In-Time Objectives: A General Approach for Specialized AI Interactions](https://mlsanity.com/paper/2510.14591/) (2025)
- [Data Analysis in the Era of Generative AI](https://mlsanity.com/paper/2409.18475/) (2024)
- [Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions](https://mlsanity.com/paper/2409.19747/) (2024)
- [Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework](https://mlsanity.com/paper/2501.13778/) (2025)
- [Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction](https://mlsanity.com/paper/2508.04842/) (2025)
- [Generative Interfaces for Language Models](https://mlsanity.com/paper/2508.19227/) (2025)
- [VisCoder2: Building Multi-Language Visualization Coding Agents](https://mlsanity.com/paper/2510.23642/) (2025)

## Abstract (arXiv)

Systems that support users in the automatic creation of visualizations must address several subtasks - understand the semantics of data, enumerate relevant visualization goals and generate visualization specifications. In this work, we pose visualization generation as a multi-stage generation problem and argue that well-orchestrated pipelines based on large language models (LLMs) such as ChatGPT/GPT-4 and image generation models (IGMs) are suitable to addressing these tasks. We present LIDA, a novel tool for generating grammar-agnostic visualizations and infographics. LIDA comprises of 4 modules - A SUMMARIZER that converts data into a rich but compact natural language summary, a GOAL EXPLORER that enumerates visualization goals given the data, a VISGENERATOR that generates, refines, executes and filters visualization code and an INFOGRAPHER module that yields data-faithful stylized graphics using IGMs. LIDA provides a python api, and a hybrid user interface (direct manipulation and multilingual natural language) for interactive chart, infographics and data story generation. Learn more about the project here - https://microsoft.github.io/lida/