# InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes arXiv: https://arxiv.org/abs/2401.05335 ยท HTML: https://mlsanity.com/paper/2401.05335/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 40 of its 46 references that are in the map. - First posted on arXiv 10 Jan 2024. ## Builds on (most cited first, 30 of 40) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022) - [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022) - [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022) - [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022) - [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022) - [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021) - [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022) - [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019) - [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023) - [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022) - [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022) - [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023) - [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021) - [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021) - [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020) - [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023) - [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023) - [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023) - [Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors](https://mlsanity.com/paper/2306.17843/) (2023) - [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023) - [HIVE: Harnessing Human Feedback for Instructional Visual Editing](https://mlsanity.com/paper/2303.09618/) (2023) - [DreamEditor: Text-Driven 3D Scene Editing with Neural Fields](https://mlsanity.com/paper/2306.13455/) (2023) - [SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields](https://mlsanity.com/paper/2211.12254/) (2022) - [Localizing Object-level Shape Variations with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.11306/) (2023) - [GAUDI: A Neural Architect for Immersive 3D Scene Generation](https://mlsanity.com/paper/2207.13751/) (2022) - [FocalDreamer: Text-driven 3D Editing via Focal-fusion Assembly](https://mlsanity.com/paper/2308.10608/) (2023) ## Built on by (most cited first, 11 of 11) - [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024) - [MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing](https://mlsanity.com/paper/2408.08000/) (2024) - [Localized Gaussian Splatting Editing with Contextual Awareness](https://mlsanity.com/paper/2408.00083/) (2024) - [DreamInsert: Zero-Shot Image-to-Video Object Insertion from A Single Image](https://mlsanity.com/paper/2503.10342/) (2025) - [FreeInsert: Disentangled Text-Guided Object Insertion in 3D Gaussian Scene without Spatial Priors](https://mlsanity.com/paper/2505.01322/) (2025) - [Towards a Training Free Approach for 3D Scene Editing](https://mlsanity.com/paper/2412.12766/) (2024) - [A Critical Synthesis of Uncertainty Quantification and Foundation Models in Monocular Depth Estimation](https://mlsanity.com/paper/2501.08188/) (2025) - [Diffusion Models are Secretly Zero-Shot 3DGS Harmonizers](https://mlsanity.com/paper/2503.06740/) (2025) - [ArCSEM: Artistic Colorization of SEM Images via Gaussian Splatting](https://mlsanity.com/paper/2410.21310/) (2024) - [Survey on Monocular Metric Depth Estimation](https://mlsanity.com/paper/2501.11841/) (2025) - [ElasticFit: Fit-Aware 3D Object Insertion via VLM Reasoning and Generative Adaptation](https://mlsanity.com/paper/2610.07460/) (2026) ## Abstract (arXiv) We introduce InseRF, a novel method for generative object insertion in the NeRF reconstructions of 3D scenes. Based on a user-provided textual description and a 2D bounding box in a reference viewpoint, InseRF generates new objects in 3D scenes. Recently, methods for 3D scene editing have been profoundly transformed, owing to the use of strong priors of text-to-image diffusion models in 3D generative modeling. Existing methods are mostly effective in editing 3D scenes via style and appearance changes or removing existing objects. Generating new objects, however, remains a challenge for such methods, which we address in this study. Specifically, we propose grounding the 3D object insertion to a 2D object insertion in a reference view of the scene. The 2D edit is then lifted to 3D using a single-view object reconstruction method. The reconstructed object is then inserted into the scene, guided by the priors of monocular depth estimation methods. We evaluate our method on various 3D scenes and provide an in-depth analysis of the proposed components. Our experiments with generative insertion of objects in several 3D scenes indicate the effectiveness of our method compared to the existing methods. InseRF is capable of controllable and 3D-consistent object insertion without requiring explicit 3D information as input. Please visit our project page at https://mohamad-shahbazi.github.io/inserf.