mlsanityData to 7 Oct 2026

Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D

arXiv 2312.02190 ↗First posted 2 Dec 2023cs.CV cs.GRcited by 59

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Diffusion Handles is a novel approach to enabling 3D object edits on diffusion images. We accomplish these edits using existing pre-trained diffusion models, and 2D image depth estimation, without any fine-tuning or 3D object retrieval. The edited results remain plausible, photo-real, and preserve object identity. Diffusion Handles address a critically missing facet of generative image based creative design, and significantly advance the state-of-the-art in generative image editing. Our key insight is to lift diffusion activations for an object to 3D using a proxy depth, 3D-transform the depth and associated activations, and project them back to image space. The diffusion process applied to the manipulated activations with identity control, produces plausible edited images showing complex 3D occlusion and lighting effects. We evaluate Diffusion Handles: quantitatively, on a large synthetic data benchmark; and qualitatively by a user study, showing our output to be more plausible, and better than prior art at both, 3D editing and identity control. Project Webpage: https://diffusionhandles.github.io/

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 47 of its 60 references that are in the map. First posted on arXiv 2 Dec 2023.

Builds on

47 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2023Segment Anythingcited by 8,158
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
34 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  4. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  5. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  8. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  9. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  10. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  12. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  13. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  14. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  15. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  16. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  17. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  18. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  19. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  20. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  21. 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 218
  22. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  23. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  24. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 193
  25. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  26. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  27. 2023StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Modelscited by 154
  28. 20193D Ken Burns Effect from a Single Imagecited by 128
  29. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  30. 2023OBJECT 3DIT: Language-guided 3D-aware Image Editingcited by 62
  31. 2021SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpaintingcited by 48
  32. 2022SceneComposer: Any-Level Semantic Image Synthesiscited by 43
  33. 2022im2nerf: Image to Neural Radiance Field in the Wildcited by 11
  34. 2023Image Harmonization with Diffusion Modelcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 26; 2026: 16.

59 papers.

Sort:

By area:

  1. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  2. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  3. 2024Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videoscited by 38
  4. 2024GeoDiffuser: Geometry-Based Image Editing with Diffusion Modelscited by 34
  5. 2024Generative Image Layer Decomposition with Visual Effectscited by 33
  6. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  7. 2024InstantDrag: Improving Interactivity in Drag-based Image Editingcited by 30
  8. 2025ObjectMover: Generative Object Movement with Video Priorcited by 30
  9. 2024Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioningcited by 26
  10. 2024ZeST: Zero-Shot Material Transfer from a Single Imagecited by 24
  11. 2024DiffUHaul: A Training-Free Method for Object Dragging in Imagescited by 22
  12. 2026Orient Anything V2: Unifying Orientation and Rotation Understandingcited by 22
38 more
  1. 2025IntrinsicEdit: Precise generative image manipulation in intrinsic spacecited by 17
  2. 2025VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priorscited by 14
  3. 2025PreciseCam: Precise Camera Control for Text-to-Image Generationcited by 13
  4. 2025Training-free Geometric Image Editing on Diffusion Modelscited by 13
  5. 20253D-Fixup: Advancing Photo Editing with 3D Priorscited by 12
  6. 2024Diffusion Models in 3D Vision: A Surveycited by 11
  7. 2024Motion Modes: What Could Happen Next?cited by 11
  8. 2025Compass Control: Multi Object Orientation Control for Text-to-Image Generationcited by 11
  9. 2024Reference-Based 3D-Aware Image Editing with Triplanescited by 9
  10. 2025I2V3D: Controllable image-to-video generation with 3D guidancecited by 9
  11. 2025GenSpace: Benchmarking Spatially-Aware Image Generationcited by 9
  12. 2025MARBLE: Material Recomposition and Blending in CLIP-Spacecited by 9
  13. 2025SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulationcited by 9
  14. 2024FlexGen: Flexible Multi-View Generation from Text and Image Inputscited by 8
  15. 2025JOG3R: Towards 3D-Consistent Video Generatorscited by 8
  16. 2025Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxycited by 8
  17. 2025BlenderFusion: 3D-Grounded Visual Editing and Generative Compositingcited by 7
  18. 2025LACONIC: A 3D Layout Adapter for Controllable Image Creationcited by 6
  19. 2024Infinite-Resolution Integral Noise Warping for Diffusion Modelscited by 5
  20. 20253D Scene Understanding Through Local Random Access Sequence Modelingcited by 5
  21. 2025MObI: Multimodal Object Inpainting Using Diffusion Modelscited by 4
  22. 2025Attention in Diffusion Model: A Surveycited by 4
  23. 2025Coupled Diffusion Sampling for Training-Free Multi-View Image Editingcited by 4
  24. 2026SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generationcited by 4
  25. 2025Pro-DG: Procedural Diffusion Guidance for Architectural Facade Generationcited by 3
  26. 2025Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Enginecited by 3
  27. 2025DragMesh: Interactive 3D Generation Made Easycited by 3
  28. 2026Ctrl&Shift: High-Quality Geometry-Aware Object Manipulation in Visual Generationcited by 3
  29. 2026Direct 3D-Aware Object Insertion via Decomposed Visual Proxiescited by 3
  30. 2024ShapeWords: Guiding Text-to-Image Synthesis with 3D Shape-Aware Promptscited by 2
  31. 2025Generative Blocks World: Moving Things Around in Picturescited by 2
  32. 2025Discovering and using Spelke segmentscited by 2
  33. 2025Neural USD: An object-centric framework for iterative editing and controlcited by 2
  34. 2026Unified 3D Scene Understanding Through Physical World Modelingcited by 2
  35. 2026Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformerscited by 1
  36. 2026PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editingcited by 1
  37. 2026GeoEdit: Geometry-Aware Object Editing via Dual-Branch Denoisingcited by 1
  38. 2024GenHeld: Generating and Editing Handheld Objectscited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Models are Geometry Critics: Single Image 3D Editing Using Pre-Trained Diffusion Priors
  2. 2023Differential Diffusion: Giving Each Pixel Its Strength
  3. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models
  4. 2023Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images
  5. 2023PAIR-Diffusion: A Comprehensive Multimodal Object-Level Image Editor
  6. 2025Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers
  7. 2024Diffusion Model-Based Image Editing: A Survey
  8. 2023State of the Art on Diffusion Models for Visual Computing