mlsanityData to 13 Oct 2026

SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction

arXiv 2310.20700 ↗First posted 31 Oct 2023cs.CVcited by 232

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recently video generation has achieved substantial progress with realistic results. Nevertheless, existing AI-generated videos are usually very short clips ("shot-level") depicting a single scene. To deliver a coherent long video ("story-level"), it is desirable to have creative transition and prediction effects across different clips. This paper presents a short-to-long video diffusion model, SEINE, that focuses on generative transition and prediction. The goal is to generate high-quality long videos with smooth and creative transitions between scenes and varying lengths of shot-level videos. Specifically, we propose a random-mask video diffusion model to automatically generate transitions based on textual descriptions. By providing the images of different scenes as inputs, combined with text-based control, our model generates transition videos that ensure coherence and visual quality. Furthermore, the model can be readily extended to various tasks such as image-to-video animation and autoregressive video prediction. To conduct a comprehensive evaluation of this new generative task, we propose three assessing criteria for smooth and creative transition: temporal consistency, semantic similarity, and video-text semantic alignment. Extensive experiments validate the effectiveness of our approach over existing methods for generative transition and prediction, enabling the creation of story-level long videos. Project page: https://vchitect.github.io/SEINE-project/ .

Key facts. Cited by 232 papers in the map (data through 13 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 36 of its 44 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

36 of 44 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  2. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  3. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  6. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  7. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov
  8. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 27

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  3. 2020Denoising Diffusion Implicit Modelscited by 9,718
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,028
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,484
  8. 2017Neural Discrete Representation Learningcited by 4,680
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,751
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,237
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,732
15 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,176
  2. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 926
  3. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 903
  4. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 725
  5. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 653
  6. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  7. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 490
  8. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 477
  9. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  10. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  11. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  12. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
  13. 2023LEO: Generative Latent Image Animator for Human Video Synthesiscited by 40
  14. 2022StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesiscited by 38
  15. 2019Gated-GAN: Adversarial Gated Networks for Multi-Collection Style Transfercited by 17

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 10; 2024: 83; 2025: 93; 2026: 46.

232 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,775
  2. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,625
  3. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 571
  4. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 539
  5. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 473
  6. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 451
  7. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 414
  8. 2025LongLive: Real-time Interactive Long Video Generationcited by 246
  9. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 245
  10. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 244
  11. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 241
  12. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
38 more
  1. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 180
  2. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  3. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 170
  4. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 152
  5. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 134
  6. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 132
  7. 2025Video World Models with Long-term Spatial Memorycited by 128
  8. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 118
  9. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 117
  10. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  11. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 104
  12. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 102
  13. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  14. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  15. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 90
  16. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  17. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 85
  18. 2025Automated Movie Generation via Multi-Agent CoT Planningcited by 83
  19. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 79
  20. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 77
  21. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 75
  22. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 73
  23. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharingcited by 71
  24. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70
  25. 2024Vlogger: Make Your Dream A Vlogcited by 70
  26. 2024ToonCrafter: Generative Cartoon Interpolationcited by 68
  27. 2024Progressive Autoregressive Video Diffusion Modelscited by 65
  28. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  29. 2025Long-Context State-Space Video World Modelscited by 61
  30. 2024COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editingcited by 59
  31. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 55
  32. 2025AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusioncited by 53
  33. 2024Follow-Your-Click: Open-domain Regional Image Animation via Short Promptscited by 52
  34. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  35. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  36. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 46
  37. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  38. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 43

50 most cited of 232 All 232 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  2. 2022Video Diffusion Models
  3. 2025The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation
  4. 2024VideoStudio: Generating Consistent-Content and Multi-Scene Videos
  5. 2025Captain Cinema: Towards Short Movie Generation
  6. 2025TokensGen: Harnessing Condensed Tokens for Long Video Generation
  7. 2023A Survey on Video Diffusion Models
  8. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models