# Character-Centered Dialogue Generation from Scene-Level Prompts

arXiv: https://arxiv.org/abs/2505.16819 · HTML: https://mlsanity.com/paper/2505.16819/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 14 of 14)

- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](https://mlsanity.com/paper/2402.17485/) (2024)
- [OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models](https://mlsanity.com/paper/2502.01061/) (2025)
- [Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer](https://mlsanity.com/paper/2412.00733/) (2024)
- [SEED-Story: Multimodal Long Story Generation with Large Language Model](https://mlsanity.com/paper/2407.08683/) (2024)
- [MoCha: Towards Movie-Grade Talking Character Synthesis](https://mlsanity.com/paper/2503.23307/) (2025)
- [MM-StoryAgent: Immersive Narrated Storybook Video Generation with a Multi-Agent Paradigm across Text, Image and Audio](https://mlsanity.com/paper/2503.05242/) (2025)
- [Let Storytelling Tell Vivid Stories: An Expressive and Fluent Multimodal Storyteller](https://mlsanity.com/paper/2403.07301/) (2024)
- [From Words to Worlds: Transforming One-line Prompt into Immersive Multi-modal Digital Stories with Communicative LLM Agent](https://mlsanity.com/paper/2406.10478/) (2024)
- [Sound of Story: Multi-modal Storytelling with Audio](https://mlsanity.com/paper/2310.19264/) (2023)
- [Dialogue Director: Bridging the Gap in Dialogue Visualization for Multimodal Storytelling](https://mlsanity.com/paper/2412.20725/) (2024)
- [Improving Visual Storytelling with Multimodal Large Language Models](https://mlsanity.com/paper/2407.02586/) (2024)

## Built on by (most cited first, 2 of 2)

- [MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration](https://mlsanity.com/paper/2602.03028/) (2026)
- [Text-Conditioned Background Generation for Editable Multi-Layer Documents](https://mlsanity.com/paper/2512.17151/) (2025)

## Abstract (arXiv)

Recent advances in scene-based video generation enable coherent visual narratives from structured prompts, yet a key aspect of storytelling -- character-driven dialogue and speech -- remains underexplored. We present a modular pipeline that transforms action-level prompts into visually and auditorily grounded dialogue, enriching scene-based storytelling with natural voice and character expression. Our method takes a pair of prompts per scene, defining the setting and character behavior. While a story generation model such as Text2Story produces the visual scene, we focus on generating expressive, character-consistent utterances grounded in both the prompts and a representative scene image. A pretrained vision-language encoder extracts high-level visual semantics, which are combined with structured prompts to guide a large language model for dialogue synthesis. To maintain contextual and emotional consistency across scenes, we introduce a Recursive Narrative Bank, a speaker-aware, temporally structured memory that accumulates each character's dialogue history. Inspired by Script Theory, this design enables dialogue that reflects evolving goals, social context, and narrative roles. Finally, we render each utterance as expressive, character-conditioned speech, producing fully voiced, multimodal video narratives. Our training-free framework generalizes across diverse story settings, providing a scalable solution for coherent, character-grounded audiovisual storytelling.