# VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual intervention

arXiv: https://arxiv.org/abs/2503.15138 · HTML: https://mlsanity.com/paper/2503.15138/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 65 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023)
- [StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text](https://mlsanity.com/paper/2403.14773/) (2024)
- [StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation](https://mlsanity.com/paper/2405.01434/) (2024)
- [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://mlsanity.com/paper/2204.03638/) (2022)

## Built on by (most cited first, 30 of 36)

- [JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization](https://mlsanity.com/paper/2511.23002/) (2025)
- [EchoShot: Multi-Shot Portrait Video Generation](https://mlsanity.com/paper/2506.15838/) (2025)
- [ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytelling](https://mlsanity.com/paper/2603.25746/) (2026)
- [ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichment](https://mlsanity.com/paper/2509.17818/) (2025)
- [NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation](https://mlsanity.com/paper/2507.11245/) (2025)
- [MAViS: A Multi-Agent Framework for Long-Sequence Video Storytelling](https://mlsanity.com/paper/2508.08487/) (2025)
- [Manifold-Aware Exploration for Reinforcement Learning in Video Generation](https://mlsanity.com/paper/2603.21872/) (2026)
- [ViMax: Agentic Video Generation](https://mlsanity.com/paper/2606.07649/) (2026)
- [CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narratives](https://mlsanity.com/paper/2605.12496/) (2026)
- [STORYANCHORS: Generating Consistent Multi-Scene Story Frames for Long-Form Narratives](https://mlsanity.com/paper/2505.08350/) (2025)
- [LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation](https://mlsanity.com/paper/2510.26412/) (2025)
- [InfinityStory: Unlimited Video Generation with World Consistency and Character-Aware Shot Transitions](https://mlsanity.com/paper/2603.03646/) (2026)
- [ShotVerse: Advancing Cinematic Camera Control for Text-Driven Multi-Shot Video Creation](https://mlsanity.com/paper/2603.11421/) (2026)
- [Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation](https://mlsanity.com/paper/2509.26555/) (2025)
- [One Sentence, One Drama: Personalized Short-Form Drama Generation via Multi-Agent Systems](https://mlsanity.com/paper/2605.22144/) (2026)
- [ReCA: Multi-Shot Long Video Extrapolation via Recursive Context Allocation](https://mlsanity.com/paper/2605.26525/) (2026)
- [CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation](https://mlsanity.com/paper/2606.09639/) (2026)
- [EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVer](https://mlsanity.com/paper/2509.12718/) (2025)
- [MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration](https://mlsanity.com/paper/2602.03028/) (2026)
- [GroundShot: Visually Consistent Multi-Shot Long Video Generation via Entity-Grounded Shot Scheduling](https://mlsanity.com/paper/2606.20799/) (2026)
- [STAR: Mitigating Cascading Errors in Spatial Reasoning via Turn-point Alignment and Segment-level DPO](https://mlsanity.com/paper/2604.00558/) (2026)
- [Camera Artist: A Multi-Agent Framework for Cinematic Language Storytelling Video Generation](https://mlsanity.com/paper/2604.09195/) (2026)
- [TIE: Time Interval Encoding for Video Generation over Events](https://mlsanity.com/paper/2605.10543/) (2026)
- [RECAP-Forcing: Retaining Content Appearances for Long Video Generation](https://mlsanity.com/paper/2608.26671/) (2026)
- [Narrative Weaver: Towards Controllable Long-Range Visual Consistency with Multi-Modal Conditioning](https://mlsanity.com/paper/2603.06688/) (2026)
- [COMIC: Agentic Sketch Comedy Generation](https://mlsanity.com/paper/2603.11048/) (2026)
- [No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs](https://mlsanity.com/paper/2606.31933/) (2026)
- [FilmWorld: Agentic Novel-to-Film Generation through Dynamic Cinematic World Modeling](https://mlsanity.com/paper/2607.19038/) (2026)
- [CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing](https://mlsanity.com/paper/2605.19484/) (2026)
- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)

## Abstract (arXiv)

Current video generation models excel at short clips but fail to produce cohesive multi-shot narratives due to disjointed visual dynamics and fractured storylines. Existing solutions either rely on extensive manual scripting/editing or prioritize single-shot fidelity over cross-scene continuity, limiting their practicality for movie-like content. We introduce VideoGen-of-Thought (VGoT), a step-by-step framework that automates multi-shot video synthesis from a single sentence by systematically addressing three core challenges: (1) Narrative Fragmentation: Existing methods lack structured storytelling. We propose dynamic storyline modeling, which first converts the user prompt into concise shot descriptions, then elaborates them into detailed, cinematic specifications across five domains (character dynamics, background continuity, relationship evolution, camera movements, HDR lighting), ensuring logical narrative progression with self-validation. (2) Visual Inconsistency: Existing approaches struggle with maintaining visual consistency across shots. Our identity-aware cross-shot propagation generates identity-preserving portrait (IPP) tokens that maintain character fidelity while allowing trait variations (expressions, aging) dictated by the storyline. (3) Transition Artifacts: Abrupt shot changes disrupt immersion. Our adjacent latent transition mechanisms implement boundary-aware reset strategies that process adjacent shots' features at transition points, enabling seamless visual flow while preserving narrative continuity. VGoT generates multi-shot videos that outperform state-of-the-art baselines by 20.4% in within-shot face consistency and 17.4% in style consistency, while achieving over 100% better cross-shot consistency and 10x fewer manual adjustments than alternatives.