# LightVA: Lightweight Visual Analytics with LLM Agent-Based Task Planning and Execution arXiv: https://arxiv.org/abs/2411.05651 ยท HTML: https://mlsanity.com/paper/2411.05651/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 44 of its 94 references that are in the map. - First posted on arXiv 8 Nov 2024. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 30 of 44) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023) - [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023) - [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023) - [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023) - [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021) - [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023) - [LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models](https://mlsanity.com/paper/2303.02927/) (2023) - [Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow](https://mlsanity.com/paper/2306.07209/) (2023) - [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023) - [ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language](https://mlsanity.com/paper/2311.01920/) (2023) - [LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning](https://mlsanity.com/paper/2401.00125/) (2023) - [Data2Vis: Automatic Generation of Data Visualizations Using Sequence to Sequence Recurrent Neural Networks](https://mlsanity.com/paper/1804.03126/) (2018) - [VizML: A Machine Learning Approach to Visualization Recommendation](https://mlsanity.com/paper/1808.04819/) (2018) - [Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries](https://mlsanity.com/paper/2312.13671/) (2023) - [DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing](https://mlsanity.com/paper/2401.10880/) (2024) - [AI4VIS: Survey on Artificial Intelligence Approaches for Data Visualization](https://mlsanity.com/paper/2102.01330/) (2021) - [DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models](https://mlsanity.com/paper/2310.03691/) (2023) - [Visualization Generation with Large Language Models: An Evaluation](https://mlsanity.com/paper/2401.11255/) (2024) - [A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization](https://mlsanity.com/paper/2012.00467/) (2020) - [Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning](https://mlsanity.com/paper/2407.20174/) (2024) - [FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow System](https://mlsanity.com/paper/1908.00681/) (2019) - [LLM4Vis: Explainable Visualization Recommendation using ChatGPT](https://mlsanity.com/paper/2310.07652/) (2023) - [LEVA: Using Large Language Models to Enhance Visual Analytics](https://mlsanity.com/paper/2403.05816/) (2024) - [ChatGPT as your Personal Data Scientist](https://mlsanity.com/paper/2305.13657/) (2023) - [Demonstration of InsightPilot: An LLM-Empowered Automated Data Exploration System](https://mlsanity.com/paper/2304.00477/) (2023) - [KG4Vis: A Knowledge Graph-Based Approach for Visualization Recommendation](https://mlsanity.com/paper/2107.12548/) (2021) ## Built on by (most cited first, 20 of 20) - [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025) - [Large Language Model-based Data Science Agent: A Survey](https://mlsanity.com/paper/2508.02744/) (2025) - [ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent](https://mlsanity.com/paper/2507.18165/) (2025) - [ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset](https://mlsanity.com/paper/2501.09349/) (2025) - [VizGenie: Toward Self-Refining, Domain-Aware Workflows for Next-Generation Scientific Visualization](https://mlsanity.com/paper/2507.21124/) (2025) - [Multi-Agent Data Visualization and Narrative Generation](https://mlsanity.com/paper/2509.00481/) (2025) - [TopoPilot: Reliable Conversational Workflow Automation for Topological Data Analysis and Visualization](https://mlsanity.com/paper/2603.25063/) (2026) - [A2P-Vis: an Analyzer-to-Presenter Agentic Pipeline for Visual Insights Generation and Reporting](https://mlsanity.com/paper/2512.22101/) (2025) - [HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization](https://mlsanity.com/paper/2606.26614/) (2026) - [A Review on Large Language Models for Visual Analytics](https://mlsanity.com/paper/2503.15176/) (2025) - [The Truth, the Whole Truth, and Nothing but the Truth: Automatic Visualization Evaluation from Reconstruction Quality](https://mlsanity.com/paper/2603.16873/) (2026) - [VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents](https://mlsanity.com/paper/2506.21582/) (2025) - [GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzles](https://mlsanity.com/paper/2506.21839/) (2025) - [Linking Heterogeneous Data with Coordinated Agent Flows for Social Media Analysis](https://mlsanity.com/paper/2510.26172/) (2025) - [From Idea to Prototype in an Afternoon: Scaffolded, AI-Assisted Rapid VA Prototyping](https://mlsanity.com/paper/2606.31311/) (2026) - [MV-Bench: Benchmarking Multimodal Large Language Models for Coordinated Multi-View Interface Construction](https://mlsanity.com/paper/2607.19910/) (2026) - [Crystalis: Progressive Nucleation and Semantic Annealing for Coordinated Multi-View Visualization Generation](https://mlsanity.com/paper/2607.24766/) (2026) - [AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysis](https://mlsanity.com/paper/2608.17834/) (2026) - [ForkSCOPE: Charting the Agentic Garden of Forking Paths](https://mlsanity.com/paper/2609.12438/) (2026) - [WidgetVA: A Widget-Centric Framework and Benchmark for Agentic Visual Analytics](https://mlsanity.com/paper/2609.24094/) (2026) ## Abstract (arXiv) Visual analytics (VA) requires analysts to iteratively propose analysis tasks based on observations and execute tasks by creating visualizations and interactive exploration to gain insights. This process demands skills in programming, data processing, and visualization tools, highlighting the need for a more intelligent, streamlined VA approach. Large language models (LLMs) have recently been developed as agents to handle various tasks with dynamic planning and tool-using capabilities, offering the potential to enhance the efficiency and versatility of VA. We propose LightVA, a lightweight VA framework that supports task decomposition, data analysis, and interactive exploration through human-agent collaboration. Our method is designed to help users progressively translate high-level analytical goals into low-level tasks, producing visualizations and deriving insights. Specifically, we introduce an LLM agent-based task planning and execution strategy, employing a recursive process involving a planner, executor, and controller. The planner is responsible for recommending and decomposing tasks, the executor handles task execution, including data analysis, visualization generation and multi-view composition, and the controller coordinates the interaction between the planner and executor. Building on the framework, we develop a system with a hybrid user interface that includes a task flow diagram for monitoring and managing the task planning process, a visualization panel for interactive data exploration, and a chat view for guiding the model through natural language instructions. We examine the effectiveness of our method through a usage scenario and an expert study.