mlsanityData to 7 Oct 2026

MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence

arXiv 2407.16655 ↗First posted 23 Jul 2024cs.CVcited by 85

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities. Homepage: https://aim-uofa.github.io/MovieDreamer/.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 51 of its 61 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

51 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation2 May
  2. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  4. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  5. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling23 Oct
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  7. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  8. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
30 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  2. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  3. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  4. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  5. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  6. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  7. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  8. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  9. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  11. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  12. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  13. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  14. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  15. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  16. 2022Learning Video Representations from Large Language Modelscited by 204
  17. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  18. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
  19. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  20. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  21. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  22. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  23. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  24. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  25. 2022Make-A-Story: Visual Memory Conditioned Consistent Story Generationcited by 76
  26. 2023Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Modelscited by 76
  27. 2022Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Modelscited by 69
  28. 2024Infinite-ID: Identity-preserved Personalization via ID-semantics Decoupling Paradigmcited by 38
  29. 2023Improved Visual Story Generation with Adaptive Context Modelingcited by 22
  30. 2023Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Controlcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 48; 2026: 24.

85 papers.

Sort:

By area:

  1. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  2. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 209
  3. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  4. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 126
  5. 2025Long Context Tuning for Video Generationcited by 110
  6. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  7. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  8. 2025Mixture of Contexts for Long Video Generationcited by 90
  9. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 72
  10. 2025Stable Video Infinity: Infinite-Length Video Generation with Error Recyclingcited by 68
  11. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 67
  12. 2025HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narrativescited by 51
38 more
  1. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  2. 2025MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narrativescited by 40
  3. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
  4. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 37
  5. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  6. 2025StoryMem: Multi-shot Long Video Storytelling with Memorycited by 35
  7. 2025VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual interventioncited by 33
  8. 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memorycited by 31
  9. 2025EchoShot: Multi-Shot Portrait Video Generationcited by 28
  10. 2025CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Modelscited by 26
  11. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  12. 2025The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generationcited by 21
  13. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 21
  14. 2024Towards Precise Scaling Laws for Video Diffusion Transformerscited by 20
  15. 2025Macro-from-Micro Planning for High-Quality and Parallelized Autoregressive Long Video Generationcited by 19
  16. 2025MAViS: A Multi-Agent Framework for Long-Sequence Video Storytellingcited by 19
  17. 2024Owl-1: Omni World Model for Consistent Long Video Generationcited by 17
  18. 2025VideoAuteur: Towards Long Narrative Video Generationcited by 17
  19. 2025Cut2Next: Generating Next Shot via In-Context Tuningcited by 15
  20. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  21. 2025Model Reveals What to Cache: Profiling-Based Feature Reuse for Video Diffusion Modelscited by 14
  22. 2025Ingredients: Blending Custom Photos with Video Diffusion Transformerscited by 13
  23. 2024Story-Iter: A Training-free Iterative Paradigm for Long Story Visualizationcited by 12
  24. 2024Video Diffusion Transformers are In-Context Learnerscited by 11
  25. 2026ViMax: Agentic Video Generationcited by 11
  26. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  27. 2025StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representationcited by 10
  28. 2025STORYANCHORS: Generating Consistent Multi-Scene Story Frames for Long-Form Narrativescited by 9
  29. 2025LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Modelcited by 9
  30. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 9
  31. 2025BIFE: Better Interaction, Fewer Errors for Minute-Long Video Generationcited by 9
  32. 2026Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPOcited by 9
  33. 2026MuSS: A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generationcited by 8
  34. 2026CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narrativescited by 8
  35. 2024DreamRunner: Fine-Grained Compositional Story-to-Video Generation with Retrieval-Augmented Motion Adaptationcited by 7
  36. 2025AnimeGamer: Infinite Anime Life Simulation with Next Game State Predictioncited by 7
  37. 2025Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generationcited by 7
  38. 2025CML-Bench: A Framework for Evaluating and Enhancing LLM-Powered Movie Scripts Generationcited by 6

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CineWeaver: Training-Free Reference-Controllable Multi-Shot Long Video Generation for Cinematic Storytelling
  2. 2026DreamShot: Personalized Storyboard Synthesis with Video Diffusion Prior
  3. 2026LongTake: Learning to Sustain Dynamics in Long-Horizon Video Generation
  4. 2023MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images
  5. 2025SeqBench: Benchmarking Sequential Narrative Generation in Text-to-Video Models
  6. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models
  7. 2024Cinematographic Camera Diffusion Model
  8. 2026DCDM: Divide-and-Conquer Diffusion Models for Consistency-Preserving Video Generation