mlsanityData to 14 Oct 2026

SinFusion: Training Diffusion Models on a Single Image or Video

arXiv 2211.11743 ↗First posted 21 Nov 2022cs.CV cs.LGcited by 58

On the map: Diffusion models → Video generation

Abstract (arXiv)

Diffusion models exhibited tremendous progress in image and video generation, exceeding GANs in quality and diversity. However, they are usually trained on very large datasets and are not naturally adapted to manipulate a given input image or video. In this paper we show how this can be resolved by training a diffusion model on a single input image or video. Our image/video-specific diffusion model (SinFusion) learns the appearance and dynamics of the single image or video, while utilizing the conditioning capabilities of diffusion models. It can solve a wide array of image/video-specific manipulation tasks. In particular, our model can learn from few frames the motion and dynamics of a single input video. It can then generate diverse new video samples of the same dynamic scene, extrapolate short videos into long ones (both forward and backward in time) and perform video upsampling. Most of these tasks are not realizable by current video-specific generation methods.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 52 of its 72 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

52 of 72 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  2. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  3. 2022Flexible Diffusion Modeling of Long Videos23 May
  4. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  5. 2022Video Diffusion Models7 Apr
  6. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  11. 2022Generative Adversarial Networkscited by 5,627
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
34 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  3. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  4. 2022A ConvNet for the 2020scited by 3,579
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  7. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  9. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  11. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  12. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  13. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  14. 2022Diffusion Models in Vision: A Surveycited by 1,015
  15. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  16. 2018Self-Attention Generative Adversarial Networkscited by 809
  17. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  18. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  19. 2018Video-to-Video Synthesiscited by 390
  20. 2016Generating Videos with Scene Dynamicscited by 376
  21. 2021GAN Inversion: A Surveycited by 367
  22. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  23. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  24. 2018Stochastic Video Generation with a Learned Priorcited by 245
  25. 2017Stochastic Variational Video Predictioncited by 234
  26. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 169
  27. 2015Delving Deeper into Convolutional Networks for Learning Video Representationscited by 148
  28. 2020Adversarial score matching and improved sampling for image generationcited by 96
  29. 2018Recycle-GAN: Unsupervised Video Retargetingcited by 89
  30. 2020Improved Techniques for Training Single-Image GANscited by 78
  31. 2020Positional Encoding as Spatial Inductive Bias in GANscited by 69
  32. 2021Drop the GAN: In Defense of Patches Nearest Neighbors as Single Image Generative Modelscited by 51
  33. 2020Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Samplecited by 43
  34. 2021Diverse Generation from a Single Video Made Possiblecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 22; 2024: 19; 2025: 10; 2026: 5.

58 papers.

Sort:

By area:

  1. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  2. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  3. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  4. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 302
  5. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  6. 2023A Survey on Video Diffusion Modelscited by 206
  7. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 183
  8. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 139
  9. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Modelscited by 133
  10. 2023GlyphControl: Glyph Conditional Control for Visual Text Generationcited by 131
  11. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  12. 2023Single Motion Diffusioncited by 65
38 more
  1. 2022SinDDM: A Single Image Denoising Diffusion Modelcited by 60
  2. 2023MotionEditor: Editing Video Motion via Content-Aware Diffusioncited by 57
  3. 2024Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditionscited by 42
  4. 2024Towards a Mechanistic Explanation of Diffusion Model Generalizationcited by 42
  5. 2024Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfercited by 31
  6. 2023Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shapecited by 29
  7. 2024Be-Your-Outpainter: Mastering Video Outpainting through Input-Specific Adaptationcited by 29
  8. 2024Hierarchical Patch Diffusion Models for High-Resolution Video Generationcited by 24
  9. 2023A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shotcited by 19
  10. 2026Code2Worlds: Empowering Coding LLMs for 4D World Generationcited by 19
  11. 2023ACDMSR: Accelerated Conditional Diffusion Models for Single Image Super-Resolutioncited by 18
  12. 2024EchoNet-Synthetic: Privacy-preserving Video Generation for Safe Medical Data Sharingcited by 18
  13. 2023Style-A-Video: Agile Diffusion for Arbitrary Text-based Video Style Transfercited by 14
  14. 2023STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Predictioncited by 12
  15. 2023Single Mesh Diffusion Models with Field Latents for Texture Generationcited by 11
  16. 2024Replication in Visual Diffusion Models: A Survey and Outlookcited by 11
  17. 2023Image Synthesis under Limited Data: A Survey and Taxonomycited by 9
  18. 2023Deep Internal Learning: Deep Learning from a Single Inputcited by 9
  19. 2024Zero-shot High-fidelity and Pose-controllable Character Animationcited by 9
  20. 2024TexTile: A Differentiable Metric for Texture Tileabilitycited by 7
  21. 2024Diffusion-Promoted HDR Video Reconstructioncited by 6
  22. 2024Segment Anything for Videos: A Systematic Surveycited by 5
  23. 2024Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicalitycited by 4
  24. 2024Robot Shape and Location Retention in Video Generation Using Diffusion Modelscited by 3
  25. 2025ShapeShifter: 3D Variations Using Multiscale and Sparse Point-Voxel Diffusioncited by 3
  26. 2025SyncSDE: A Probabilistic Framework for Diffusion Synchronizationcited by 3
  27. 2024Content-aware Tile Generation using Exterior Boundary Inpaintingcited by 2
  28. 2024Unicorn: Unified Neural Image Compression with One Number Reconstructioncited by 2
  29. 2025Generative Video Bi-flowcited by 2
  30. 2025TriTex: Learning Texture from a Single Mesh via Triplane Semantic Featurescited by 2
  31. 2023Infusion: internal diffusion for inpainting of dynamic textures and complex motioncited by 1
  32. 2023Leveraging Graph Diffusion Models for Network Refinement Taskscited by 1
  33. 2024Neural Differential Appearance Equationscited by 1
  34. 2025HALO: Human-Aligned End-to-end Image Retargeting with Layered Transformationscited by 1
  35. 2025Deep Learning Empowered Super-Resolution: A Comprehensive Survey and Future Prospectscited by 1
  36. 2026StructDiff: A Structure-Preserving and Spatially Controllable Diffusion Model for Single-Image Generationcited by 1
  37. 2026Efficient and Training-Free Single-Image Diffusion Modelscited by 1
  38. 2024JVID: Joint Video-Image Diffusion for Visual-Quality and Temporal-Consistency in Video Generationcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation
  2. 2024Accelerating Video Diffusion Models via Distribution Matching
  3. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  4. 2024OSV: One Step is Enough for High-Quality Image to Video Generation
  5. 2022VIDM: Video Implicit Diffusion Models
  6. 2025Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
  7. 2022Wavelet Diffusion Models are fast and scalable Image Generators
  8. 2024SF-V: Single Forward Video Generation Model