mlsanityData to 7 Oct 2026

OBJECT 3DIT: Language-guided 3D-aware Image Editing

arXiv 2307.11073 ↗First posted 20 Jul 2023cs.CV cs.AI cs.GRcited by 62

Not on the map. Why?

Cited by 21 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing image editing tools, while powerful, typically disregard the underlying 3D geometry from which the image is projected. As a result, edits made using these tools may become detached from the geometry and lighting conditions that are at the foundation of the image formation process. In this work, we formulate the newt ask of language-guided 3D-aware editing, where objects in an image should be edited according to a language instruction in context of the underlying 3D scene. To promote progress towards this goal, we release OBJECT: a dataset consisting of 400K editing examples created from procedurally generated 3D scenes. Each example consists of an input image, editing instruction in language, and the edited image. We also introduce 3DIT : single and multi-task models for four editing tasks. Our models show impressive abilities to understand the 3D composition of entire scenes, factoring in surrounding objects, surfaces, lighting conditions, shadows, and physically-plausible object configurations. Surprisingly, training on only synthetic scenes from OBJECT, editing capabilities of 3DIT generalize to real-world images.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 69 of its 85 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

69 of 85 references.

References · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2023Segment Anythingcited by 8,158
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  6. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  9. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  12. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  13. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  14. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  15. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  16. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  17. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  18. 2017Deep Image Priorcited by 1,097
  19. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  20. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  21. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  22. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  23. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  24. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  25. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  26. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  27. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  28. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  29. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  30. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  31. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  32. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  33. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 393
  34. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  35. 2022Kubric: A scalable dataset generatorcited by 373
  36. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  37. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  38. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 17; 2025: 25; 2026: 17.

62 papers.

Sort:

By area:

  1. 2023Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for nowcited by 61
  2. 2023Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3Dcited by 59
  3. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priorscited by 53
  4. 2024Image Sculpting: Precise Object Editing with 3D Geometry Controlcited by 45
  5. 2024Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Modelscited by 42
  6. 2025OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Datacited by 35
  7. 2024GeoDiffuser: Geometry-Based Image Editing with Diffusion Modelscited by 34
  8. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  9. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  10. 2025ObjectMover: Generative Object Movement with Video Priorcited by 30
  11. 2024Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversioncited by 27
  12. 20243DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splattingcited by 25
38 more
  1. 2024ZeST: Zero-Shot Material Transfer from a Single Imagecited by 24
  2. 2024General Place Recognition Survey: Towards Real-World Autonomycited by 23
  3. 2024ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusioncited by 16
  4. 2025VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priorscited by 14
  5. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 14
  6. 2024Customizing Text-to-Image Diffusion with Object Viewpoint Controlcited by 13
  7. 2024Localized Gaussian Splatting Editing with Contextual Awarenesscited by 13
  8. 2024Diffusion Models are Geometry Critics: Single Image 3D Editing Using Pre-Trained Diffusion Priorscited by 12
  9. 20253D-Fixup: Advancing Photo Editing with 3D Priorscited by 12
  10. 2025Compass Control: Multi Object Orientation Control for Text-to-Image Generationcited by 11
  11. 2024TrAME: Trajectory-Anchored Multi-View Editing for Text-Guided 3D Gaussian Splatting Manipulationcited by 10
  12. 2024Can CLIP Count Stars? An Empirical Study on Quantity Bias in CLIPcited by 9
  13. 2025I2V3D: Controllable image-to-video generation with 3D guidancecited by 9
  14. 2025SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulationcited by 9
  15. 2025MirrorVerse: Pushing Diffusion Models to Realistically Reflect the Worldcited by 8
  16. 2025Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxycited by 8
  17. 2023Lasagna: Layered Score Distillation for Disentangled Object Relightingcited by 7
  18. 20253D Object Manipulation in a Single Image using Generative Modelscited by 7
  19. 2025BlenderFusion: 3D-Grounded Visual Editing and Generative Compositingcited by 7
  20. 2025LACONIC: A 3D Layout Adapter for Controllable Image Creationcited by 6
  21. 2025The Promise of RL for Autoregressive Image Editingcited by 6
  22. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  23. 20253D Scene Understanding Through Local Random Access Sequence Modelingcited by 5
  24. 2024EraseDraw: Learning to Draw Step-by-Step via Erasing Objects from Imagescited by 4
  25. 2025MObI: Multimodal Object Inpainting Using Diffusion Modelscited by 4
  26. 2025Pro-DG: Procedural Diffusion Guidance for Architectural Facade Generationcited by 3
  27. 2025Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Enginecited by 3
  28. 2026Ctrl&Shift: High-Quality Geometry-Aware Object Manipulation in Visual Generationcited by 3
  29. 2026Direct 3D-Aware Object Insertion via Decomposed Visual Proxiescited by 3
  30. 2025Generative Blocks World: Moving Things Around in Picturescited by 2
  31. 2025Controllable 3D Placement of Objects with Scene-Aware Diffusion Modelscited by 2
  32. 2025AutoScape: Geometry-Consistent Long-Horizon Scene Generationcited by 2
  33. 2025Neural USD: An object-centric framework for iterative editing and controlcited by 2
  34. 2026EZBlender: Efficient 3D Editing with Plan-and-ReAct Agentcited by 2
  35. 2026Unified 3D Scene Understanding Through Physical World Modelingcited by 2
  36. 2026Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformerscited by 1
  37. 2026PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editingcited by 1
  38. 2026GeoEdit: Geometry-Aware Object Editing via Dual-Branch Denoisingcited by 1

50 most cited of 62 All 62 (JSON)