# MM-StoryAgent: Immersive Narrated Storybook Video Generation with a Multi-Agent Paradigm across Text, Image and Audio

arXiv: https://arxiv.org/abs/2503.05242 · HTML: https://mlsanity.com/paper/2503.05242/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation](https://mlsanity.com/paper/2405.01434/) (2024)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion](https://mlsanity.com/paper/2407.12899/) (2024)
- [MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generation](https://mlsanity.com/paper/2406.05690/) (2024)
- [On The Open Prompt Challenge In Conditional Audio Generation](https://mlsanity.com/paper/2311.00897/) (2023)

## Built on by (most cited first, 27 of 27)

- [AniMaker: Multi-Agent Animated Storytelling with MCTS-Driven Clip Generation](https://mlsanity.com/paper/2506.10540/) (2025)
- [PreGenie: An Agentic Framework for High-quality Visual Presentation Generation](https://mlsanity.com/paper/2505.21660/) (2025)
- [MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation](https://mlsanity.com/paper/2602.23969/) (2026)
- [Web Fraud Attacks Against LLM-Driven Multi-Agent Systems](https://mlsanity.com/paper/2509.01211/) (2025)
- [ViStoryBench: Comprehensive Benchmark Suite for Story Visualization](https://mlsanity.com/paper/2505.24862/) (2025)
- [AnimeAgent: Is the Multi-Agent via Image-to-Video models a Good Disney Storytelling Artist?](https://mlsanity.com/paper/2602.20664/) (2026)
- [GroundShot: Visually Consistent Multi-Shot Long Video Generation via Entity-Grounded Shot Scheduling](https://mlsanity.com/paper/2606.20799/) (2026)
- [PersonaVlog: Personalized Multimodal Vlog Generation with Multi-Agent Collaboration and Iterative Self-Correction](https://mlsanity.com/paper/2508.13602/) (2025)
- [AudioStory: Generating Long-Form Narrative Audio with Large Language Models](https://mlsanity.com/paper/2508.20088/) (2025)
- [Better Call CineCrew: Consistent Ultra-Long Narrative-to-Film Generation](https://mlsanity.com/paper/2609.07720/) (2026)
- [Character-Centered Dialogue Generation from Scene-Level Prompts](https://mlsanity.com/paper/2505.16819/) (2025)
- [VisionCreator: A Native Visual-Generation Agentic Model with Understanding, Thinking, Planning and Creation](https://mlsanity.com/paper/2603.02681/) (2026)
- [LogiStory: A Logic-Aware Framework for Multi-Image Story Visualization](https://mlsanity.com/paper/2603.28082/) (2026)
- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [FilmWorld: Agentic Novel-to-Film Generation through Dynamic Cinematic World Modeling](https://mlsanity.com/paper/2607.19038/) (2026)
- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)
- [CreAgentive: An Agent Workflow Driven Multi-Category Creative Generation Engine](https://mlsanity.com/paper/2509.26461/) (2025)
- [DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Models](https://mlsanity.com/paper/2512.01686/) (2025)
- [VisionCreator-R1: A Reflection-Enhanced Native Visual-Generation Agentic Model](https://mlsanity.com/paper/2603.08812/) (2026)
- [Self-Reasoning Agentic Framework for Narrative Product Grid-Collage Generation](https://mlsanity.com/paper/2604.16958/) (2026)
- [Crayotter: Traceable Multi-Agent Workflows for Long-Form Video Editing](https://mlsanity.com/paper/2606.07636/) (2026)
- [GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzles](https://mlsanity.com/paper/2506.21839/) (2025)
- [KathaTrace: Diagnosing Semantic Trajectory Collapse in Generated Visual Narratives](https://mlsanity.com/paper/2607.01312/) (2026)
- [Dramarrator: Object-Based Audio Editing for Audio Drama Production from Books](https://mlsanity.com/paper/2608.08349/) (2026)
- [FRAMEWORKERS: A Dynamic Multi-Agent Framework for AI-Generated Video Production](https://mlsanity.com/paper/2608.29814/) (2026)
- [CogenPVG: Cognitive-Enhanced Reflective Multi-Agent Framework for Persuasive Video Generation](https://mlsanity.com/paper/2609.25821/) (2026)
- [VIS-Ground: Video Interactive Storytelling with Contextual Grounding](https://mlsanity.com/paper/2610.09326/) (2026)

## Abstract (arXiv)

The rapid advancement of large language models (LLMs) and artificial intelligence-generated content (AIGC) has accelerated AI-native applications, such as AI-based storybooks that automate engaging story production for children. However, challenges remain in improving story attractiveness, enriching storytelling expressiveness, and developing open-source evaluation benchmarks and frameworks. Therefore, we propose and opensource MM-StoryAgent, which creates immersive narrated video storybooks with refined plots, role-consistent images, and multi-channel audio. MM-StoryAgent designs a multi-agent framework that employs LLMs and diverse expert tools (generative models and APIs) across several modalities to produce expressive storytelling videos. The framework enhances story attractiveness through a multi-stage writing pipeline. In addition, it improves the immersive storytelling experience by integrating sound effects with visual, music and narrative assets. MM-StoryAgent offers a flexible, open-source platform for further development, where generative modules can be substituted. Both objective and subjective evaluation regarding textual story quality and alignment between modalities validate the effectiveness of our proposed MM-StoryAgent system. The demo and source code are available.