mlsanityData to 14 Oct 2026

DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editing

arXiv 2403.12002 ↗First posted 18 Mar 2024cs.CV cs.AIcited by 28

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-driven diffusion-based video editing presents a unique challenge not encountered in image editing literature: establishing real-world motion. Unlike existing video editing approaches, here we focus on score distillation sampling to circumvent the standard reverse diffusion process and initiate optimization from videos that already exhibit natural motion. Our analysis reveals that while video score distillation can effectively introduce new content indicated by target text, it can also cause significant structure and motion deviation. To counteract this, we propose to match space-time self-similarities of the original video and the edited video during the score distillation. Thanks to the use of score distillation, our approach is model-agnostic, which can be applied for both cascaded and non-cascaded video diffusion frameworks. Through extensive comparisons with leading methods, our approach demonstrates its superiority in altering appearances while accurately preserving the original structure and motion.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 60 of its 71 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

60 of 71 references.

Closest ancestors · 22

Same direction, newest first

  1. 2023MotionCrafter: One-Shot Motion Customization of Diffusion Models8 Dec
  2. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motion7 Dec
  3. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models1 Dec
  4. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  5. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
  6. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  7. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing9 Oct
  8. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models2 Oct
  9. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  10. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  11. 2023ModelScope Text-to-Video Technical Report12 Aug
  12. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet26 Jul
10 more
  1. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  3. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  5. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  6. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  7. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  8. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  10. 2022Video Diffusion Models7 Apr

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
26 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  8. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  10. 2021Grounded Language-Image Pre-trainingcited by 1,156
  11. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  12. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  13. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  14. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  15. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 460
  16. 2023Emergent Correspondence from Image Diffusioncited by 446
  17. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  18. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  19. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  20. 2019Style Transfer by Relaxed Optimal Transport and Self-Similaritycited by 168
  21. 2022Diffusion-based Image Translation using Disentangled Style and Content Representationcited by 149
  22. 2023TextMesh: Generation of Realistic 3D Meshes From Text Promptscited by 135
  23. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  24. 2023Delta Denoising Scorecited by 121
  25. 2023Contrastive Denoising Score for Text-guided Latent Diffusion Image Editingcited by 49
  26. 2023Collaborative Score Distillation for Consistent Visual Synthesiscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 6.

28 papers.

Sort:

By area:

  1. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  2. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  3. 2025FlowAlign: Trajectory-Regularized, Inversion-Free Flow-based Image Editingcited by 32
  4. 2024Motion Inversion for Video Customizationcited by 30
  5. 2024SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Devicecited by 27
  6. 2025FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarioscited by 25
  7. 2025DreamRelation: Relation-Centric Video Customizationcited by 23
  8. 2026FastVMT: Eliminating Redundancy in Video Motion Transfercited by 23
  9. 2025Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generationcited by 15
  10. 2025VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priorscited by 14
  11. 2025Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfercited by 14
  12. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 14
16 more
  1. 2025Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videoscited by 14
  2. 2025H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Modelscited by 13
  3. 2025TransPixeler: Advancing Text-to-Video Generation with Transparencycited by 12
  4. 2025Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewardscited by 11
  5. 2026SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generationcited by 10
  6. 2025Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometrycited by 7
  7. 2025Taming Flow-based I2V Models for Creative Video Editingcited by 5
  8. 2026DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learningcited by 4
  9. 2026Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editingcited by 3
  10. 2026Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generationcited by 3
  11. 2024Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuningcited by 2
  12. 2026Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshootingcited by 2
  13. 2024PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Datasetcited by 0
  14. 2025Interactive Video Generation via Domain Adaptationcited by 0
  15. 2025VIDMP3: Video Editing by Representing Motion with Pose and Position Priorscited by 0
  16. 2025Motion Marionette: Rethinking Rigid Motion Transfer via Prior Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Dreamix: Video Diffusion Models are General Video Editors
  2. 2024DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation
  3. 2024Video Editing via Factorized Diffusion Distillation
  4. 2026Beyond Consistency: Preserving Temporal Structure in Zero-Shot Video Editing
  5. 2024Diffusion Model-Based Video Editing: A Survey
  6. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing
  7. 2026Adaptive Video Distillation: Mitigating Oversaturation and Temporal Collapse in Few-Step Generation
  8. 2024MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion