mlsanityData to 7 Oct 2026

Structure and Content-Guided Video Synthesis with Diffusion Models

arXiv 2302.03011 ↗First posted 6 Feb 2023cs.CVcited by 589

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

Text-guided generative diffusion models unlock powerful image creation and editing tools. While these have been extended to video generation, current approaches that edit the content of existing footage while retaining structure require expensive re-training for every input or rely on error-prone propagation of image edits across frames. In this work, we present a structure and content-guided video diffusion model that edits videos based on visual or textual descriptions of the desired output. Conflicts between user-provided content edits and structure representations occur due to insufficient disentanglement between the two aspects. As a solution, we show that training on monocular depth estimates with varying levels of detail provides control over structure and content fidelity. Our model is trained jointly on images and videos which also exposes explicit control of temporal consistency through a novel guidance method. Our experiments demonstrate a wide variety of successes; fine-grained control over output characteristics, customization based on a few reference images, and a strong user preference towards results by our model.

Key facts. Cited by 589 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 60 of its 69 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

60 of 69 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Scalable Diffusion Models with Transformers19 Dec
  2. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr
  3. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2022Video Diffusion Modelscited by 2,280
  6. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  11. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  12. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  13. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  14. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  15. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  16. 2022Human Motion Diffusion Modelcited by 1,062
  17. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  18. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  19. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  20. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  21. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  22. 2021Score-based Generative Modeling in Latent Spacecited by 634
  23. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  24. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  25. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  26. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  27. 2018Video-to-Video Synthesiscited by 388
  28. 2016Generating Videos with Scene Dynamicscited by 376
  29. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  30. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  31. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 275
  32. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  33. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  34. 2019Few-shot Video-to-Video Synthesiscited by 196
  35. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
  36. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  37. 2022Generating Long Videos of Dynamic Scenescited by 113
  38. 2017Coherent Online Video Style Transfercited by 96

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 166; 2024: 210; 2025: 162; 2026: 51.

589 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  7. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  8. 2024Genie: Generative Interactive Environmentscited by 706
  9. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  10. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  11. 2023ModelScope Text-to-Video Technical Reportcited by 651
  12. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
38 more
  1. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  2. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  3. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  4. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  5. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  6. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  7. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  8. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  9. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  10. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  11. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  12. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  13. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  14. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  15. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  16. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  17. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  18. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  19. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  20. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 276
  21. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  22. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  23. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 242
  24. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  25. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  26. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  27. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  28. 2023A Survey on Video Diffusion Modelscited by 203
  29. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  30. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  31. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  32. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  33. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  34. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  35. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  36. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  37. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  38. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161

50 most cited of 589 All 589 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Dreamix: Video Diffusion Models are General Video Editors
  2. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models
  3. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  4. 2023MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance
  5. 2024Temporally Consistent Object Editing in Videos using Extended Attention
  6. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  7. 2026DCDM: Divide-and-Conquer Diffusion Models for Consistency-Preserving Video Generation
  8. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models