# From Words to Worlds: Transforming One-line Prompt into Immersive Multi-modal Digital Stories with Communicative LLM Agent

arXiv: https://arxiv.org/abs/2406.10478 · HTML: https://mlsanity.com/paper/2406.10478/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Synthesizing Physical Character-Scene Interactions](https://mlsanity.com/paper/2302.00883/) (2023)
- [StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation](https://mlsanity.com/paper/2209.06192/) (2022)
- [Story Realization: Expanding Plot Events into Sentences](https://mlsanity.com/paper/1909.03480/) (2019)
- [AdaptNet: Policy Adaptation for Physics-Based Character Control](https://mlsanity.com/paper/2310.00239/) (2023)
- [Persona-Guided Planning for Controlling the Protagonist's Persona in Story Generation](https://mlsanity.com/paper/2204.10703/) (2022)
- [Story-to-Motion: Synthesizing Infinite and Controllable Character Animation from Long Text](https://mlsanity.com/paper/2311.07446/) (2023)
- [The Five-Dollar Model: Generating Game Maps and Sprites from Sentence Embeddings](https://mlsanity.com/paper/2308.04052/) (2023)
- [M3Act: Learning from Synthetic Human Group Activities](https://mlsanity.com/paper/2306.16772/) (2023)
- [The Importance of Multimodal Emotion Conditioning and Affect Consistency for Embodied Conversational Agents](https://mlsanity.com/paper/2309.15311/) (2023)
- [On the Equivalency, Substitutability, and Flexibility of Synthetic Data](https://mlsanity.com/paper/2403.16244/) (2024)

## Built on by (most cited first, 4 of 4)

- [LLM Agents for Education: Advances and Applications](https://mlsanity.com/paper/2503.11733/) (2025)
- [The Effects of Embodiment and Personality Expression on Learning in LLM-based Educational Agents](https://mlsanity.com/paper/2407.10993/) (2024)
- [Better Call CineCrew: Consistent Ultra-Long Narrative-to-Film Generation](https://mlsanity.com/paper/2609.07720/) (2026)
- [Character-Centered Dialogue Generation from Scene-Level Prompts](https://mlsanity.com/paper/2505.16819/) (2025)

## Abstract (arXiv)

Digital storytelling, essential in entertainment, education, and marketing, faces challenges in production scalability and flexibility. The StoryAgent framework, introduced in this paper, utilizes Large Language Models and generative tools to automate and refine digital storytelling. Employing a top-down story drafting and bottom-up asset generation approach, StoryAgent tackles key issues such as manual intervention, interactive scene orchestration, and narrative consistency. This framework enables efficient production of interactive and consistent narratives across multiple modalities, democratizing content creation and enhancing engagement. Our results demonstrate the framework's capability to produce coherent digital stories without reference videos, marking a significant advancement in automated digital storytelling.