# Frame-Level Captions for Long Video Generation with Complex Multi Scenes

arXiv: https://arxiv.org/abs/2505.20827 · HTML: https://mlsanity.com/paper/2505.20827/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 46 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 30 of 42)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [Open-Sora Plan: Open-Source Large Video Generation Model](https://mlsanity.com/paper/2412.00131/) (2024)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text](https://mlsanity.com/paper/2403.14773/) (2024)
- [StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation](https://mlsanity.com/paper/2405.01434/) (2024)
- [SkyReels-V2: Infinite-length Film Generative Model](https://mlsanity.com/paper/2504.13074/) (2025)
- [Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions](https://mlsanity.com/paper/2111.10337/) (2021)
- [Autoregressive Diffusion Models](https://mlsanity.com/paper/2110.02037/) (2021)
- [Unified Video Action Model](https://mlsanity.com/paper/2503.00200/) (2025)
- [FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling](https://mlsanity.com/paper/2310.15169/) (2023)
- [Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model](https://mlsanity.com/paper/2502.10248/) (2025)
- [Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising](https://mlsanity.com/paper/2305.18264/) (2023)
- [FIFO-Diffusion: Generating Infinite Videos from Text without Training](https://mlsanity.com/paper/2405.11473/) (2024)

## Built on by (most cited first, 1 of 1)

- [PhysVid: Physics Aware Local Conditioning for Generative Video Models](https://mlsanity.com/paper/2603.26285/) (2026)

## Abstract (arXiv)

Generating long videos that can show complex stories, like movie scenes from scripts, has great promise and offers much more than short clips. However, current methods that use autoregression with diffusion models often struggle because their step-by-step process naturally leads to a serious error accumulation (drift). Also, many existing ways to make long videos focus on single, continuous scenes, making them less useful for stories with many events and changes. This paper introduces a new approach to solve these problems. First, we propose a novel way to annotate datasets at the frame-level, providing detailed text guidance needed for making complex, multi-scene long videos. This detailed guidance works with a Frame-Level Attention Mechanism to make sure text and video match precisely. A key feature is that each part (frame) within these windows can be guided by its own distinct text prompt. Our training uses Diffusion Forcing to provide the model with the ability to handle time flexibly. We tested our approach on difficult VBench 2.0 benchmarks ("Complex Plots" and "Complex Landscapes") based on the WanX2.1-T2V-1.3B model. The results show our method is better at following instructions in complex, changing scenes and creates high-quality long videos. We plan to share our dataset annotation methods and trained models with the research community. Project page: https://zgctroy.github.io/frame-level-captions .