# DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editing

arXiv: https://arxiv.org/abs/2403.12002 · HTML: https://mlsanity.com/paper/2403.12002/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 60 of its 71 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 60)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)

## Built on by (most cited first, 28 of 28)

- [Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuning](https://mlsanity.com/paper/2506.05207/) (2025)
- [Reangle-A-Video: 4D Video Generation as Video-to-Video Translation](https://mlsanity.com/paper/2503.09151/) (2025)
- [FlowAlign: Trajectory-Regularized, Inversion-Free Flow-based Image Editing](https://mlsanity.com/paper/2505.23145/) (2025)
- [Motion Inversion for Video Customization](https://mlsanity.com/paper/2403.20193/) (2024)
- [SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Device](https://mlsanity.com/paper/2412.10494/) (2024)
- [FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios](https://mlsanity.com/paper/2505.03730/) (2025)
- [DreamRelation: Relation-Centric Video Customization](https://mlsanity.com/paper/2503.07602/) (2025)
- [FastVMT: Eliminating Redundancy in Video Motion Transfer](https://mlsanity.com/paper/2602.05551/) (2026)
- [Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generation](https://mlsanity.com/paper/2501.01425/) (2025)
- [VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors](https://mlsanity.com/paper/2503.01107/) (2025)
- [Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfer](https://mlsanity.com/paper/2503.17350/) (2025)
- [Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising](https://mlsanity.com/paper/2503.20782/) (2025)
- [Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos](https://mlsanity.com/paper/2507.12646/) (2025)
- [H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Models](https://mlsanity.com/paper/2504.10567/) (2025)
- [TransPixeler: Advancing Text-to-Video Generation with Transparency](https://mlsanity.com/paper/2501.03006/) (2025)
- [Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewards](https://mlsanity.com/paper/2506.15684/) (2025)
- [SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generation](https://mlsanity.com/paper/2604.03723/) (2026)
- [Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometry](https://mlsanity.com/paper/2506.13697/) (2025)
- [Taming Flow-based I2V Models for Creative Video Editing](https://mlsanity.com/paper/2509.21917/) (2025)
- [DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning](https://mlsanity.com/paper/2603.12257/) (2026)
- [Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editing](https://mlsanity.com/paper/2601.16296/) (2026)
- [Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation](https://mlsanity.com/paper/2604.02289/) (2026)
- [Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuning](https://mlsanity.com/paper/2412.02114/) (2024)
- [Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting](https://mlsanity.com/paper/2604.21776/) (2026)
- [PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset](https://mlsanity.com/paper/2412.12877/) (2024)
- [Interactive Video Generation via Domain Adaptation](https://mlsanity.com/paper/2505.24253/) (2025)
- [VIDMP3: Video Editing by Representing Motion with Pose and Position Priors](https://mlsanity.com/paper/2510.12069/) (2025)
- [Motion Marionette: Rethinking Rigid Motion Transfer via Prior Guidance](https://mlsanity.com/paper/2511.19909/) (2025)

## Abstract (arXiv)

Text-driven diffusion-based video editing presents a unique challenge not encountered in image editing literature: establishing real-world motion. Unlike existing video editing approaches, here we focus on score distillation sampling to circumvent the standard reverse diffusion process and initiate optimization from videos that already exhibit natural motion. Our analysis reveals that while video score distillation can effectively introduce new content indicated by target text, it can also cause significant structure and motion deviation. To counteract this, we propose to match space-time self-similarities of the original video and the edited video during the score distillation. Thanks to the use of score distillation, our approach is model-agnostic, which can be applied for both cascaded and non-cascaded video diffusion frameworks. Through extensive comparisons with leading methods, our approach demonstrates its superiority in altering appearances while accurately preserving the original structure and motion.