mlsanityData to 7 Oct 2026

Gaussian Grouping: Segment and Edit Anything in 3D Scenes

arXiv 2312.00732 ↗First posted 1 Dec 2023cs.CV cs.AIcited by 387

Not on the map. Why?

Abstract (arXiv)

The recent Gaussian Splatting achieves high-quality and real-time novel-view synthesis of the 3D scenes. However, it is solely concentrated on the appearance and geometry modeling, while lacking in fine-grained object-level scene understanding. To address this issue, we propose Gaussian Grouping, which extends Gaussian Splatting to jointly reconstruct and segment anything in open-world 3D scenes. We augment each Gaussian with a compact Identity Encoding, allowing the Gaussians to be grouped according to their object instance or stuff membership in the 3D scene. Instead of resorting to expensive 3D labels, we supervise the Identity Encodings during the differentiable rendering by leveraging the 2D mask predictions by Segment Anything Model (SAM), along with introduced 3D spatial consistency regularization. Compared to the implicit NeRF representation, we show that the discrete and grouped 3D Gaussians can reconstruct, segment and edit anything in 3D with high visual quality, fine granularity and efficiency. Based on Gaussian Grouping, we further propose a local Gaussian Editing scheme, which shows efficacy in versatile scene editing applications, including 3D object removal, inpainting, colorization, style transfer and scene recomposition. Our code and models are at https://github.com/lkeab/gaussian-grouping.

Key facts. Cited by 387 papers in the map (data through 7 Oct 2026). Builds on 58 of its 68 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

58 of 68 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023Segment Anythingcited by 8,158
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  10. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  11. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  12. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
38 more
  1. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
  2. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  3. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  4. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  5. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  6. 2023LERF: Language Embedded Radiance Fieldscited by 538
  7. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  8. 2023Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splattingcited by 470
  9. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  10. 2023Segment Anything in High Qualitycited by 373
  11. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  12. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  13. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 345
  14. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  15. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  16. 2023Text-to-3D using Gaussian Splattingcited by 299
  17. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  18. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  19. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  20. 2019Differentiable Surface Splatting for Point-based Geometry Processingcited by 238
  21. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  22. 2021Editing Conditional Radiance Fieldscited by 222
  23. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 216
  24. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  25. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  26. 2021Point-Based Neural Rendering with Per-View Optimizationcited by 188
  27. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  28. 2023SAM3D: Segment Anything in 3D Scenescited by 155
  29. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  30. 2021Editable Free-viewpoint Video Using a Layered Neural Representationcited by 148
  31. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  32. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  33. 2021Unsupervised Discovery of Object Radiance Fieldscited by 100
  34. 2023Anything-3D: Towards Single-view Anything Reconstruction in the Wildcited by 91
  35. 2023Neural Point Catacaustics for Novel-View Synthesis of Reflectionscited by 81
  36. 2022Object-Compositional Neural Implicit Surfacescited by 74
  37. 2022DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Imagescited by 69
  38. 2021CoNeRF: Controllable Neural Radiance Fieldscited by 65

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 104; 2025: 145; 2026: 135.

387 papers.

Sort:

By area:

  1. 2024A Survey on 3D Gaussian Splattingcited by 221
  2. 2024VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Realitycited by 175
  3. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 141
  4. 2024Recent Advances in 3D Gaussian Splattingcited by 130
  5. 2024Revising Densification in Gaussian Splattingcited by 127
  6. 20243D Gaussian as a New Era: A Surveycited by 102
  7. 2024Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videoscited by 96
  8. 2025PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyondcited by 91
  9. 2024SGS-SLAM: Semantic Gaussian Splatting For Neural Dense SLAMcited by 86
  10. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 81
  11. 20243D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunitiescited by 77
  12. 2024CG-SLAM: Efficient Dense RGB-D SLAM in a Consistent Uncertainty-aware 3D Gaussian Fieldcited by 61
38 more
  1. 2024Click-Gaussian: Interactive Segmentation to Any 3D Gaussianscited by 58
  2. 2024FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimallycited by 57
  3. 2024InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Priorcited by 55
  4. 2024GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplanecited by 53
  5. 2024VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstructioncited by 50
  6. 2024Texture-GS: Disentangling the Geometry and Texture for 3D Gaussian Splatting Editingcited by 47
  7. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 46
  8. 2024Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editingcited by 44
  9. 2024SemGauss-SLAM: Dense Semantic Gaussian Splatting SLAMcited by 41
  10. 2024Gaga: Group Any Gaussians via 3D-aware Memory Bankcited by 40
  11. 2024Gaussian Splatting: 3D Reconstruction and Novel View Synthesis, a Reviewcited by 40
  12. 2024Textured Gaussians for Enhanced 3D Scene Appearance Modelingcited by 40
  13. 2025LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPScited by 40
  14. 2025Deformable Beta Splattingcited by 38
  15. 2024LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenescited by 36
  16. 2024InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perceptioncited by 35
  17. 20254D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Modelscited by 35
  18. 2024FastLGS: Speeding up Language Embedded Gaussians with Feature Grid Mappingcited by 34
  19. 20243D Vision-Language Gaussian Splattingcited by 30
  20. 2024Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splattingcited by 30
  21. 2024SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussianscited by 30
  22. 2024SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decompositioncited by 29
  23. 2025Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splattingcited by 29
  24. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  25. 2024DepthLab: From Partial to Completecited by 27
  26. 2024Feat2GS: Probing Visual Foundation Models with Gaussian Splattingcited by 26
  27. 2024LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priorscited by 25
  28. 2024GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splattingcited by 25
  29. 2025ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splattingcited by 24
  30. 2024StylizedGS: Controllable Stylization for 3D Gaussian Splattingcited by 23
  31. 2025Lifting by Gaussians: A Simple, Fast and Flexible Method for 3D Instance Segmentationcited by 23
  32. 20253D Gaussian Inpainting with Depth-Guided Cross-View Consistencycited by 23
  33. 2024Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentationcited by 22
  34. 2024Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotioncited by 22
  35. 2024EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splattingcited by 22
  36. 2024MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflectionscited by 21
  37. 2024ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their Self-Supervised Pretrainingcited by 21
  38. 2025AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360deg Unbounded Scene Inpaintingcited by 21

50 most cited of 387 All 387 (JSON)