# Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos

arXiv: https://arxiv.org/abs/2403.13044 · HTML: https://mlsanity.com/paper/2403.13044/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 69 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 30 of 52)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Universal Guidance for Diffusion Models](https://mlsanity.com/paper/2302.07121/) (2023)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)

## Built on by (most cited first, 30 of 38)

- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [GeoDiffuser: Geometry-Based Image Editing with Diffusion Models](https://mlsanity.com/paper/2404.14403/) (2024)
- [Generative Image Layer Decomposition with Visual Effects](https://mlsanity.com/paper/2411.17864/) (2024)
- [Learning Action and Reasoning-Centric Image Editing from Videos and Simulations](https://mlsanity.com/paper/2407.03471/) (2024)
- [InstantDrag: Improving Interactivity in Drag-based Image Editing](https://mlsanity.com/paper/2409.08857/) (2024)
- [Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models](https://mlsanity.com/paper/2406.09292/) (2024)
- [ObjectMover: Generative Object Movement with Video Prior](https://mlsanity.com/paper/2503.08037/) (2025)
- [Pathways on the Image Manifold: Image Editing via Video Generation](https://mlsanity.com/paper/2411.16819/) (2024)
- [DiffUHaul: A Training-Free Method for Object Dragging in Images](https://mlsanity.com/paper/2406.01594/) (2024)
- [VividDream: Generating 3D Scene with Ambient Dynamics](https://mlsanity.com/paper/2405.20334/) (2024)
- [LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos](https://mlsanity.com/paper/2405.13722/) (2024)
- [ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motions](https://mlsanity.com/paper/2506.03107/) (2025)
- [Face Anonymization Made Simple](https://mlsanity.com/paper/2411.00762/) (2024)
- [IntrinsicEdit: Precise generative image manipulation in intrinsic space](https://mlsanity.com/paper/2505.08889/) (2025)
- [VINCIE: Unlocking In-context Image Editing from Video](https://mlsanity.com/paper/2506.10941/) (2025)
- [VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors](https://mlsanity.com/paper/2503.01107/) (2025)
- [FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors](https://mlsanity.com/paper/2501.08225/) (2025)
- [Instruction-based Image Manipulation by Watching How Things Move](https://mlsanity.com/paper/2412.12087/) (2024)
- [3D-Fixup: Advancing Photo Editing with 3D Priors](https://mlsanity.com/paper/2505.10566/) (2025)
- [BlenderFusion: 3D-Grounded Visual Editing and Generative Compositing](https://mlsanity.com/paper/2506.17450/) (2025)
- [Coupled Diffusion Sampling for Training-Free Multi-View Image Editing](https://mlsanity.com/paper/2510.14981/) (2025)
- [Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Engine](https://mlsanity.com/paper/2511.13713/) (2025)
- [Over++: Generative Video Compositing for Layer Interaction Effects](https://mlsanity.com/paper/2512.19661/) (2025)
- [BlobCtrl: Taming Controllable Blob for Element-level Image Editing](https://mlsanity.com/paper/2503.13434/) (2025)
- [Neural USD: An object-centric framework for iterative editing and control](https://mlsanity.com/paper/2510.23956/) (2025)
- [HALO: Human-Aligned End-to-end Image Retargeting with Layered Transformations](https://mlsanity.com/paper/2504.03026/) (2025)
- [CrimEdit: Controllable Editing for Counterfactual Object Removal, Insertion, and Movement](https://mlsanity.com/paper/2509.23708/) (2025)
- [LooseRoPE: Content-aware Attention Manipulation for Semantic Harmonization](https://mlsanity.com/paper/2601.05127/) (2026)
- [Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformers](https://mlsanity.com/paper/2602.08388/) (2026)

## Abstract (arXiv)

We propose a generative model that, given a coarsely edited image, synthesizes a photorealistic output that follows the prescribed layout. Our method transfers fine details from the original image and preserve the identity of its parts. Yet, it adapts it to the lighting and context defined by the new layout. Our key insight is that videos are a powerful source of supervision for this task: objects and camera motions provide many observations of how the world changes with viewpoint, lighting, and physical interactions. We construct an image dataset in which each sample is a pair of source and target frames extracted from the same video at randomly chosen time intervals. We warp the source frame toward the target using two motion models that mimic the expected test-time user edits. We supervise our model to translate the warped image into the ground truth, starting from a pretrained diffusion model. Our model design explicitly enables fine detail transfer from the source frame to the generated image, while closely following the user-specified layout. We show that by using simple segmentations and coarse 2D manipulations, we can synthesize a photorealistic edit faithful to the user's input while addressing second-order effects like harmonizing the lighting and physical interactions between edited objects.