mlsanityData to 7 Oct 2026

Lumiere: A Space-Time Diffusion Model for Video Generation

arXiv 2401.12945 ↗First posted 23 Jan 2024cs.CVcited by 450

On the map: Diffusion models → Video generation

Abstract (arXiv)

We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution -- an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. We demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation.

Key facts. Cited by 450 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 50 of its 73 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

50 of 73 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Inflation with Diffusion: Efficient Temporal Adaptation for Text-to-Video Super-Resolution18 Jan
  2. 2023Photorealistic Video Generation with Diffusion Models11 Dec
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  4. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  5. 2023ModelScope Text-to-Video Technical Report12 Aug
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  7. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  9. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 40

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
28 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  6. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  8. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  9. 2022Diffusion Models in Vision: A Surveycited by 1,008
  10. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  12. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  13. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  14. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  15. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  16. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  17. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  18. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  19. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  20. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  21. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 193
  22. 2019Style Transfer by Relaxed Optimal Transport and Self-Similaritycited by 167
  23. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 161
  24. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  25. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  26. 2020Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domainscited by 96
  27. 2023Matryoshka Diffusion Modelscited by 70
  28. 2023Breathing Life Into Sketches Using Text-to-Video Priorscited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 161; 2025: 159; 2026: 130.

450 papers.

Sort:

By area:

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  3. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  4. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  5. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  6. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  7. 2024One Step Diffusion via Shortcut Modelscited by 373
  8. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  9. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  10. 2024Navigation World Modelscited by 262
  11. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  12. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
38 more
  1. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  2. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  3. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  4. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  5. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 185
  6. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  7. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  8. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  9. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  10. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 160
  11. 2024Behavior Generation with Latent Actionscited by 159
  12. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  13. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  14. 2025Do generative video models understand physical principles?cited by 151
  15. 2026Advancing Open-source World Modelscited by 144
  16. 2024Consistency Models Made Easycited by 140
  17. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  18. 2024Dynamical Regimes of Diffusion Modelscited by 133
  19. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  20. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
  21. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  22. 2025Long Context Tuning for Video Generationcited by 110
  23. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  24. 2024GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creationcited by 99
  25. 2024Loong: Generating Minute-level Long Videos with Autoregressive Language Modelscited by 99
  26. 2025Controllable Video Generation: A Surveycited by 93
  27. 2025Mixture of Contexts for Long Video Generationcited by 90
  28. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  29. 2024Video as the New Language for Real-World Decision Makingcited by 86
  30. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  31. 2024Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusioncited by 76
  32. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  33. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 70
  34. 2024Fisher Flow Matching for Generative Modeling over Discrete Datacited by 68
  35. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 68
  36. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  37. 2025HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characterscited by 64
  38. 2024Image Conductor: Precision Control for Interactive Video Synthesiscited by 62

50 most cited of 450 All 450 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Video Diffusion Models
  2. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
  3. 2025LumosFlow: Motion-Guided Long Video Generation
  4. 2026Efficient Video Diffusion Models: Advancements and Challenges
  5. 2025Lumina-Video: Efficient and Flexible Video Generation with Multi-scale Next-DiT
  6. 2024S2DM: Sector-Shaped Diffusion Models for Video Generation
  7. 2026UniVid: Pyramid Diffusion Model for High Quality Video Generation
  8. 2024Grid Diffusion Models for Text-to-Video Generation