# Generative Object Insertion in Gaussian Splatting with a Multi-View Diffusion Model arXiv: https://arxiv.org/abs/2409.16938 ยท HTML: https://mlsanity.com/paper/2409.16938/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. - Builds on 42 of its 54 references that are in the map. - First posted on arXiv 25 Sep 2024. ## Placed in - Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/) ## Builds on (most cited first, 30 of 42) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022) - [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021) - [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019) - [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020) - [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023) - [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022) - [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021) - [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023) - [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023) - [Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16322/) (2023) - [GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting](https://mlsanity.com/paper/2311.14521/) (2023) - [LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching](https://mlsanity.com/paper/2311.11284/) (2023) - [SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model](https://mlsanity.com/paper/2212.05034/) (2022) - [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023) - [DreamEditor: Text-Driven 3D Scene Editing with Neural Fields](https://mlsanity.com/paper/2306.13455/) (2023) - [Multi-Scale 3D Gaussian Splatting for Anti-Aliased Rendering](https://mlsanity.com/paper/2311.17089/) (2023) - [Text-to-3D with Classifier Score Distillation](https://mlsanity.com/paper/2310.19415/) (2023) - [NeRF-Art: Text-Driven Neural Radiance Fields Stylization](https://mlsanity.com/paper/2212.08070/) (2022) - [RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos](https://mlsanity.com/paper/2401.12592/) (2024) - [HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance](https://mlsanity.com/paper/2305.18766/) (2023) - [HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting](https://mlsanity.com/paper/2311.17061/) (2023) - [TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts](https://mlsanity.com/paper/2401.14828/) (2024) - [SKED: Sketch-guided Text-based 3D Editing](https://mlsanity.com/paper/2303.10735/) (2023) - [ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields](https://mlsanity.com/paper/2402.00864/) (2024) - [Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model](https://mlsanity.com/paper/2310.07222/) (2023) ## Built on by (most cited first, 6 of 6) - [BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting](https://mlsanity.com/paper/2504.09097/) (2025) - [ContraGS: Codebook-Condensed and Trainable Gaussian Splatting for Fast, Memory-Efficient Reconstruction](https://mlsanity.com/paper/2509.03775/) (2025) - [Diffusion Models are Secretly Zero-Shot 3DGS Harmonizers](https://mlsanity.com/paper/2503.06740/) (2025) - [DreamJourney: Perpetual View Generation with Video Diffusion Models](https://mlsanity.com/paper/2506.17705/) (2025) - [Enhancing Small-Scale Dataset Expansion with Triplet-Connection-based Sample Re-Weighting](https://mlsanity.com/paper/2508.07723/) (2025) - [Z-Order Transformer for Feed-Forward Gaussian Splatting](https://mlsanity.com/paper/2605.13465/) (2026) ## Abstract (arXiv) Generating and inserting new objects into 3D content is a compelling approach for achieving versatile scene recreation. Existing methods, which rely on SDS optimization or single-view inpainting, often struggle to produce high-quality results. To address this, we propose a novel method for object insertion in 3D content represented by Gaussian Splatting. Our approach introduces a multi-view diffusion model, dubbed MVInpainter, which is built upon a pre-trained stable video diffusion model to facilitate view-consistent object inpainting. Within MVInpainter, we incorporate a ControlNet-based conditional injection module to enable controlled and more predictable multi-view generation. After generating the multi-view inpainted results, we further propose a mask-aware 3D reconstruction technique to refine Gaussian Splatting reconstruction from these sparse inpainted views. By leveraging these fabricate techniques, our approach yields diverse results, ensures view-consistent and harmonious insertions, and produces better object quality. Extensive experiments demonstrate that our approach outperforms existing methods.