# From Data to Story: Towards Automatic Animated Data Video Creation with LLM-based Multi-Agent Systems arXiv: https://arxiv.org/abs/2408.03876 ยท HTML: https://mlsanity.com/paper/2408.03876/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 21 of its 43 references that are in the map. - First posted on arXiv 7 Aug 2024. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 21 of 21) - [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022) - [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023) - [LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models](https://mlsanity.com/paper/2303.02927/) (2023) - [Towards Natural Language Interfaces for Data Visualization: A Survey](https://mlsanity.com/paper/2109.03506/) (2021) - [Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study](https://mlsanity.com/paper/2404.17136/) (2024) - [Towards Natural Language-Based Visualization Authoring](https://mlsanity.com/paper/2208.10947/) (2022) - [Data Player: Automatic Generation of Data Videos with Narration-Animation Interplay](https://mlsanity.com/paper/2308.04703/) (2023) - [Where Are We So Far? Understanding Data Storytelling Tools from the Perspective of Human-AI Collaboration](https://mlsanity.com/paper/2309.15723/) (2023) - [InkSight: Leveraging Sketch Interaction for Documenting Chart Findings in Computational Notebooks](https://mlsanity.com/paper/2307.07922/) (2023) - [Reviving Static Charts into Live Charts](https://mlsanity.com/paper/2309.02967/) (2023) - [Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytelling](https://mlsanity.com/paper/2304.08366/) (2023) - [Wakey-Wakey: Animate Text by Mimicking Characters in a GIF](https://mlsanity.com/paper/2308.00224/) (2023) - [WonderFlow: Narration-Centric Design of Animated Data Videos](https://mlsanity.com/paper/2308.04040/) (2023) - [Visual Data Analysis with Task-based Recommendations](https://mlsanity.com/paper/2205.03183/) (2022) - [DIVI: Dynamically Interactive Visualization](https://mlsanity.com/paper/2310.17814/) (2023) - [DMiner: Dashboard Design Mining and Recommendation](https://mlsanity.com/paper/2209.01599/) (2022) - [Graph Exploration with Embedding-Guided Layouts](https://mlsanity.com/paper/2208.13699/) (2022) - [GeoCamera: Telling Stories in Geographic Visualizations with Camera Movements](https://mlsanity.com/paper/2303.06460/) (2023) - [NotePlayer: Engaging Jupyter Notebooks for Dynamic Presentation of Analytical Processes](https://mlsanity.com/paper/2408.01101/) (2024) ## Built on by (most cited first, 16 of 16) - [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025) - [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025) - [Code2Video: A Code-centric Paradigm for Educational Video Generation](https://mlsanity.com/paper/2510.01174/) (2025) - [Large Language Model-based Data Science Agent: A Survey](https://mlsanity.com/paper/2508.02744/) (2025) - [InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions](https://mlsanity.com/paper/2503.04110/) (2025) - [Agentic Visualization: Extracting Agent-based Design Patterns from Visualization Systems](https://mlsanity.com/paper/2505.19101/) (2025) - [Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework](https://mlsanity.com/paper/2501.13778/) (2025) - [Watermarking Large Language Models and the Generated Content: Opportunities and Challenges](https://mlsanity.com/paper/2410.19096/) (2024) - [MDSF: Context-Aware Multi-Dimensional Data Storytelling Framework based on Large language Model](https://mlsanity.com/paper/2501.01014/) (2025) - [Reflection on Data Storytelling Tools in the Generative AI Era from the Human-AI Collaboration Perspective](https://mlsanity.com/paper/2503.02631/) (2025) - [Multi-Agent Multimodal Large Language Model Framework for Automated Interpretation of Fuel Efficiency Analytics in Public Transportation](https://mlsanity.com/paper/2511.13476/) (2025) - [Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistant](https://mlsanity.com/paper/2601.17622/) (2026) - [InconLens: Interactive Visual Diagnosis of Behavioral Inconsistencies in LLM-based Agentic Systems](https://mlsanity.com/paper/2603.28106/) (2026) - [DATAREEL: Automated Data-Driven Video Story Generation with Animations](https://mlsanity.com/paper/2604.25220/) (2026) - [Data Journalist Agent: Transforming Data into Verifiable Multimodal Stories](https://mlsanity.com/paper/2606.11176/) (2026) - [Chart Deception in Vision-Language Models: From Vulnerability to Mitigation](https://mlsanity.com/paper/2607.22600/) (2026) ## Abstract (arXiv) Creating data stories from raw data is challenging due to humans' limited attention spans and the need for specialized skills. Recent advancements in large language models (LLMs) offer great opportunities to develop systems with autonomous agents to streamline the data storytelling workflow. Though multi-agent systems have benefits such as fully realizing LLM potentials with decomposed tasks for individual agents, designing such systems also faces challenges in task decomposition, performance optimization for sub-tasks, and workflow design. To better understand these issues, we develop Data Director, an LLM-based multi-agent system designed to automate the creation of animated data videos, a representative genre of data stories. Data Director interprets raw data, breaks down tasks, designs agent roles to make informed decisions automatically, and seamlessly integrates diverse components of data videos. A case study demonstrates Data Director's effectiveness in generating data videos. Throughout development, we have derived lessons learned from addressing challenges, guiding further advancements in autonomous agents for data storytelling. We also shed light on future directions for global optimization, human-in-the-loop design, and the application of advanced multi-modal LLMs.