# Image Sculpting: Precise Object Editing with 3D Geometry Control

arXiv: https://arxiv.org/abs/2401.01702 · HTML: https://mlsanity.com/paper/2401.01702/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 88 references that are in the map.
- First posted on arXiv 2 Jan 2024.

## Builds on (most cited first, 30 of 62)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)

## Built on by (most cited first, 30 of 45)

- [PhysMotion: Physics-Grounded Dynamics From a Single Image](https://mlsanity.com/paper/2411.17189/) (2024)
- [LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis](https://mlsanity.com/paper/2412.15214/) (2024)
- [Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos](https://mlsanity.com/paper/2403.13044/) (2024)
- [Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models](https://mlsanity.com/paper/2406.09292/) (2024)
- [ObjectMover: Generative Object Movement with Video Prior](https://mlsanity.com/paper/2503.08037/) (2025)
- [Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion](https://mlsanity.com/paper/2403.14617/) (2024)
- [VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors](https://mlsanity.com/paper/2503.01107/) (2025)
- [Customizing Text-to-Image Diffusion with Object Viewpoint Control](https://mlsanity.com/paper/2404.12333/) (2024)
- [Training-free Geometric Image Editing on Diffusion Models](https://mlsanity.com/paper/2507.23300/) (2025)
- [3D-Fixup: Advancing Photo Editing with 3D Priors](https://mlsanity.com/paper/2505.10566/) (2025)
- [ORIDa: Object-centric Real-world Image Composition Dataset](https://mlsanity.com/paper/2506.08964/) (2025)
- [I2V3D: Controllable image-to-video generation with 3D guidance](https://mlsanity.com/paper/2503.09733/) (2025)
- [GenSpace: Benchmarking Spatially-Aware Image Generation](https://mlsanity.com/paper/2505.24870/) (2025)
- [Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy](https://mlsanity.com/paper/2506.22432/) (2025)
- [SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation](https://mlsanity.com/paper/2511.16666/) (2025)
- [DragGaussian: Enabling Drag-style Manipulation on 3D Gaussian Representation](https://mlsanity.com/paper/2405.05800/) (2024)
- [3D Object Manipulation in a Single Image using Generative Models](https://mlsanity.com/paper/2501.12935/) (2025)
- [BlenderFusion: 3D-Grounded Visual Editing and Generative Compositing](https://mlsanity.com/paper/2506.17450/) (2025)
- [Smart Vision-Language Reasoners](https://mlsanity.com/paper/2407.04212/) (2024)
- [MObI: Multimodal Object Inpainting Using Diffusion Models](https://mlsanity.com/paper/2501.03173/) (2025)
- [Attention in Diffusion Model: A Survey](https://mlsanity.com/paper/2504.03738/) (2025)
- [SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation](https://mlsanity.com/paper/2602.23359/) (2026)
- [Anti-Inpainting: A Proactive Defense Approach against Malicious Diffusion-based Inpainters under Unknown Conditions](https://mlsanity.com/paper/2505.13023/) (2025)
- [Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Engine](https://mlsanity.com/paper/2511.13713/) (2025)
- [Ctrl&Shift: High-Quality Geometry-Aware Object Manipulation in Visual Generation](https://mlsanity.com/paper/2602.11440/) (2026)
- [Direct 3D-Aware Object Insertion via Decomposed Visual Proxies](https://mlsanity.com/paper/2606.06601/) (2026)
- [BlobCtrl: Taming Controllable Blob for Element-level Image Editing](https://mlsanity.com/paper/2503.13434/) (2025)
- [Generative Blocks World: Moving Things Around in Pictures](https://mlsanity.com/paper/2506.20703/) (2025)
- [PromptFlare: Prompt-Generalized Defense via Cross-Attention Decoy in Diffusion-Based Inpainting](https://mlsanity.com/paper/2508.16217/) (2025)
- [GeoDiffusion: A Training-Free Framework for Accurate 3D Geometric Conditioning in Image Generation](https://mlsanity.com/paper/2510.22337/) (2025)

## Abstract (arXiv)

We present Image Sculpting, a new framework for editing 2D images by incorporating tools from 3D geometry and graphics. This approach differs markedly from existing methods, which are confined to 2D spaces and typically rely on textual instructions, leading to ambiguity and limited control. Image Sculpting converts 2D objects into 3D, enabling direct interaction with their 3D geometry. Post-editing, these objects are re-rendered into 2D, merging into the original image to produce high-fidelity results through a coarse-to-fine enhancement process. The framework supports precise, quantifiable, and physically-plausible editing options such as pose editing, rotation, translation, 3D composition, carving, and serial addition. It marks an initial step towards combining the creative freedom of generative models with the precision of graphics pipelines.