# 3DSceneEditor: Controllable 3D Scene Editing with Gaussian Splatting

arXiv: https://arxiv.org/abs/2412.01583 · HTML: https://mlsanity.com/paper/2412.01583/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 79 of its 99 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Builds on (most cited first, 30 of 79)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering](https://mlsanity.com/paper/2312.00109/) (2023)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model](https://mlsanity.com/paper/2311.06214/) (2023)

## Built on by (most cited first, 22 of 22)

- [Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation](https://mlsanity.com/paper/2601.10880/) (2026)
- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [In-Context Meta LoRA Generation](https://mlsanity.com/paper/2501.17635/) (2025)
- [Inpaint360GS: Efficient Object-Aware 3D Inpainting via Gaussian Splatting for 360deg Scenes](https://mlsanity.com/paper/2511.06457/) (2025)
- [EditCast3D: Single-Frame-Guided 3D Editing with Video Propagation and View Selection](https://mlsanity.com/paper/2510.13652/) (2025)
- [GM-MoE: Low-Light Enhancement with Gated-Mechanism Mixture-of-Experts](https://mlsanity.com/paper/2503.07417/) (2025)
- [VecSet-Edit: Unleashing Pre-trained LRM for Mesh Editing from Single Image](https://mlsanity.com/paper/2602.04349/) (2026)
- [3D Scene Change Modeling With Consistent Multi-View Aggregation](https://mlsanity.com/paper/2512.22830/) (2025)
- [Variation-aware Flexible 3D Gaussian Editing](https://mlsanity.com/paper/2602.11638/) (2026)
- [TRACE: High-Fidelity 3D Scene Editing via Tangible Reconstruction and Geometry-Aligned Contextual Video Masking](https://mlsanity.com/paper/2604.01207/) (2026)
- [Sparse Code Uplifting for Efficient 3D Language Gaussian Splatting](https://mlsanity.com/paper/2605.13600/) (2026)
- [Physics-Aware 3D Gaussian Editing for Driving Scene Generation](https://mlsanity.com/paper/2605.25373/) (2026)
- [Self-Evolving Deep Research via Joint Generation and Evaluation](https://mlsanity.com/paper/2606.04507/) (2026)
- [GP-GS: Gaussian Processes Densification for 3D Gaussian Splatting](https://mlsanity.com/paper/2502.02283/) (2025)
- [Error-Driven Scene Editing for 3D Grounding in Large Language Models](https://mlsanity.com/paper/2511.14086/) (2025)
- [EditVerse3D: High-Quality 3D Object Editing with Region-Aware Learning](https://mlsanity.com/paper/2607.07187/) (2026)
- [Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstruction](https://mlsanity.com/paper/2609.39960/) (2026)
- [AdvSplat: Adversarial Attacks on Feed-Forward Gaussian Splatting Models](https://mlsanity.com/paper/2603.23686/) (2026)
- [Improving Multimodal Reasoning via Worst Dimension Optimization](https://mlsanity.com/paper/2606.07801/) (2026)
- [AnchorSplat: Fast and Structure Consistent Detail Synthesis for Gaussian Splatting](https://mlsanity.com/paper/2607.01290/) (2026)
- [ChromaGS: Text-Driven Semantic Editing of 4D Gaussian Avatars](https://mlsanity.com/paper/2610.03441/) (2026)
- [RePrior: Measurement-Reconciled Prior Correction for Sparse-View CBCT Reconstruction](https://mlsanity.com/paper/2610.14102/) (2026)

## Abstract (arXiv)

The creation of 3D scenes has traditionally been both labor-intensive and costly, requiring designers to meticulously configure 3D assets and environments. Recent advancements in generative AI, including text-to-3D and image-to-3D methods, have dramatically reduced the complexity and cost of this process. However, current techniques for editing complex 3D scenes continue to rely on generally interactive multi-step, 2D-to-3D projection methods and diffusion-based techniques, which often lack precision in control and hamper interactive-rate performance. In this work, we propose ***3DSceneEditor***, a fully 3D-based paradigm for interactive-rate, precise editing of intricate 3D scenes using Gaussian Splatting. Unlike conventional methods, 3DSceneEditor operates through a streamlined 3D pipeline, enabling direct Gaussian-based manipulation for efficient, high-quality edits based on input prompts. The proposed framework (i) integrates a pre-trained instance segmentation model for semantic labeling; (ii) employs a zero-shot grounding approach with CLIP to align target objects with user prompts; and (iii) applies scene modifications, such as object addition, repositioning, recoloring, replacing, and removal--directly on Gaussians. Extensive experimental results show that 3DSceneEditor surpasses existing state-of-the-art techniques in terms of both editing precision and efficiency, establishing a new benchmark for efficient and interactive 3D scene customization.