mlsanityData to 14 Oct 2026

Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators

arXiv 2401.18085 ↗First posted 31 Jan 2024cs.CVcited by 39

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Diffusion models are capable of generating impressive images conditioned on text descriptions, and extensions of these models allow users to edit images at a relatively coarse scale. However, the ability to precisely edit the layout, position, pose, and shape of objects in images with diffusion models is still difficult. To this end, we propose motion guidance, a zero-shot technique that allows a user to specify dense, complex motion fields that indicate where each pixel in an image should move. Motion guidance works by steering the diffusion sampling process with the gradients through an off-the-shelf optical flow network. Specifically, we design a guidance loss that encourages the sample to have the desired motion, as estimated by a flow network, while also being visually similar to the source image. By simultaneously sampling from a diffusion model and guiding the sample to have low guidance loss, we can obtain a motion-edited image. We demonstrate that our technique works on complex motions and produces high quality edits of real and generated images.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 37 of its 42 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

37 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Readout Guidance: Learning Control from Diffusion Features4 Dec
  2. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models5 Jul
  3. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing26 Jun
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  6. 2022UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance28 Oct
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 30

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  12. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
18 more
  1. 2022Video Diffusion Modelscited by 2,305
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  4. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  5. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  6. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  7. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 881
  8. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 583
  9. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  10. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 327
  11. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  12. 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 219
  13. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusionscited by 110
  14. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 108
  15. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  16. 2023Generative Image Dynamicscited by 94
  17. 2021Comparing Correspondences: Video Prediction with Correspondence-wise Lossescited by 16
  18. 2023Self-Supervised Motion Magnification by Backpropagating Through Optical Flowcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 17; 2026: 3.

39 papers.

Sort:

By area:

  1. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 191
  2. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  3. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  4. 2024TFG: Unified Training-Free Guidance for Diffusion Modelscited by 107
  5. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 66
  6. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  7. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  8. 2024MotionCraft: Physics-based Zero-Shot Video Generationcited by 48
  9. 2024Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videoscited by 38
  10. 2024GeoDiffuser: Geometry-Based Image Editing with Diffusion Modelscited by 34
  11. 2024DragAPart: Learning a Part-Level Motion Prior for Articulated Objectscited by 32
  12. 2024InstantDrag: Improving Interactivity in Drag-based Image Editingcited by 31
27 more
  1. 2025ObjectMover: Generative Object Movement with Video Priorcited by 30
  2. 2024Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamicscited by 28
  3. 2025Generative Physical AI in Vision: A Surveycited by 24
  4. 2024Factorized Diffusion: Perceptual Illusions by Noise Decompositioncited by 23
  5. 2024LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videoscited by 19
  6. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  7. 2024Optical-Flow Guided Prompt Optimization for Coherent Video Generationcited by 17
  8. 2024DreamMover: Leveraging the Prior of Diffusion Models for Image Interpolation with Large Motioncited by 16
  9. 2025PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Modelcited by 15
  10. 2025VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priorscited by 14
  11. 2025Training-free Geometric Image Editing on Diffusion Modelscited by 13
  12. 2025Compass Control: Multi Object Orientation Control for Text-to-Image Generationcited by 11
  13. 2024Text-to-Image Synthesis: A Decade Surveycited by 9
  14. 2024MikuDance: Animating Character Art with Mixed Motion Dynamicscited by 7
  15. 2025Training-free Dense-Aligned Diffusion Guidance for Modular Conditional Image Synthesiscited by 6
  16. 2025GPS as a Control Signal for Image Generationcited by 5
  17. 2025EditCLIP: Representation Learning for Image Editingcited by 5
  18. 2025POEM: Precise Object-level Editing via MLLM controlcited by 4
  19. 2025Training-Free Reward-Guided Image Editing via Trajectory Optimal Controlcited by 4
  20. 2025Coupled Diffusion Sampling for Training-Free Multi-View Image Editingcited by 4
  21. 2024TimeRewind: Rewinding Time with Image-and-Events Video Diffusioncited by 3
  22. 2025GeCo: Evaluating Geometric Consistency for Video Generation via Motion and Structurecited by 3
  23. 2026LightCtrl: Training-free Controllable Video Relightingcited by 3
  24. 2026Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformerscited by 1
  25. 2025MotionDiff: Training-free Zero-shot Interactive Motion Editing via Flow-assisted Multi-view Diffusioncited by 0
  26. 2025Interactive Video Generation via Domain Adaptationcited by 0
  27. 2026LightMover: Generative Light Movement with Color and Intensity Controlscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Filter-Guided Diffusion for Controllable Image Generation
  2. 2024MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion
  3. 2023Pix2Video: Video Editing using Image Diffusion
  4. 2026Reproducing DragDiffusion: Interactive Point-Based Editing with Diffusion Models
  5. 2026Drag within Prior Distribution: Text-Conditioned Point-Based Image Editing within Distribution Constraints
  6. 2024Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing
  7. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models
  8. 2025A Diffusion-Based Framework for Occluded Object Movement