mlsanityData to 14 Oct 2026

MagicVideo: Efficient Video Generation With Latent Diffusion Models

arXiv 2211.11018 ↗First posted 20 Nov 2022cs.CVcited by 465

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present an efficient text-to-video generation framework based on latent diffusion models, termed MagicVideo. MagicVideo can generate smooth video clips that are concordant with the given text descriptions. Due to a novel and efficient 3D U-Net design and modeling video distributions in a low-dimensional space, MagicVideo can synthesize video clips with 256x256 spatial resolution on a single GPU card, which takes around 64x fewer computations than the Video Diffusion Models (VDM) in terms of FLOPs. In specific, unlike existing works that directly train video models in the RGB space, we use a pre-trained VAE to map video clips into a low-dimensional latent space and learn the distribution of videos' latent codes via a diffusion model. Besides, we introduce two new designs to adapt the U-Net denoiser trained on image tasks to video data: a frame-wise lightweight adaptor for the image-to-video distribution adjustment and a directed temporal attention module to capture temporal dependencies across frames. Thus, we can exploit the informative weights of convolution operators from a text-to-image model for accelerating video training. To ameliorate the pixel dithering in the generated videos, we also propose a novel VideoVAE auto-encoder for better RGB reconstruction. We conduct extensive experiments and demonstrate that MagicVideo can generate high-quality video clips with either realistic or imaginary content. Refer to urlhttps://magicvideo.github.io/# for more examples.

Key facts. Cited by 465 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 49 of its 57 references that are in the map. First posted on arXiv 20 Nov 2022.

Builds on

49 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  2. 2022Flexible Diffusion Modeling of Long Videos23 May
  3. 2022Video Diffusion Models7 Apr
  4. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2016Layer Normalizationcited by 5,669
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
33 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  6. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  7. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  10. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  12. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  13. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  14. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  15. 2022Denoising Diffusion Restoration Modelscited by 963
  16. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  17. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  18. 2021Score-based Generative Modeling in Latent Spacecited by 636
  19. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  20. 2016Generating Videos with Scene Dynamicscited by 376
  21. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  22. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  23. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  24. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  25. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  26. 2022Frozen CLIP Models are Efficient Video Learnerscited by 182
  27. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
  28. 2022Understanding The Robustness in Vision Transformerscited by 144
  29. 2016Video Pixel Networkscited by 108
  30. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
  31. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  32. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  33. 2022MagicMix: Semantic Mixing with Diffusion Modelscited by 63

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 131; 2024: 179; 2025: 112; 2026: 40.

465 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  7. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  8. 2023ModelScope Text-to-Video Technical Reportcited by 658
  9. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  10. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  11. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  12. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
38 more
  1. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  2. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  5. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  6. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  7. 2023Learning Interactive Real-World Simulatorscited by 458
  8. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  9. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  10. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  11. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  12. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  13. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  14. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  15. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  16. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  17. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  18. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  19. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  20. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 302
  21. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  22. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 266
  23. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  24. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  25. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 233
  26. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  27. 2023A Survey on Video Diffusion Modelscited by 206
  28. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
  29. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
  30. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  31. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  32. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  33. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  34. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 183
  35. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  36. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 174
  37. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  38. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 171

50 most cited of 465 All 465 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate
  2. 2026Paris 2.0: A Decentralized Diffusion Model for Video Generation
  3. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
  4. 2024xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
  5. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Models
  6. 2024Latte: Latent Diffusion Transformer for Video Generation
  7. 2026RhymeFlow: Training-Free Acceleration for Video Generation with Asynchronous Denoising Flow Scheduling
  8. 2025EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation