# Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting

arXiv: https://arxiv.org/abs/2503.14029 · HTML: https://mlsanity.com/paper/2503.14029/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 53 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Mip-Splatting: Alias-free 3D Gaussian Splatting](https://mlsanity.com/paper/2311.16493/) (2023)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting](https://mlsanity.com/paper/2310.10642/) (2023)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [GaussianPro: 3D Gaussian Splatting with Progressive Propagation](https://mlsanity.com/paper/2402.14650/) (2024)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Point-Based Neural Rendering with Per-View Optimization](https://mlsanity.com/paper/2109.02369/) (2021)
- [EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything](https://mlsanity.com/paper/2312.00863/) (2023)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [Panoptic Lifting for 3D Scene Understanding with Neural Fields](https://mlsanity.com/paper/2212.09802/) (2022)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing](https://mlsanity.com/paper/2403.17822/) (2024)
- [Pixel-GS: Density Control with Pixel-aware Gradient for 3D Gaussian Splatting](https://mlsanity.com/paper/2403.15530/) (2024)
- [GARField: Group Anything with Radiance Fields](https://mlsanity.com/paper/2401.09419/) (2024)

## Built on by (most cited first, 30 of 31)

- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [CAGS: Open-Vocabulary 3D Scene Understanding with Context-Aware Gaussian Splatting](https://mlsanity.com/paper/2504.11893/) (2025)
- [COS3D: Collaborative Open-Vocabulary 3D Segmentation](https://mlsanity.com/paper/2510.20238/) (2025)
- [Zero-Shot Visual Grounding in 3D Gaussians via View Retrieval](https://mlsanity.com/paper/2509.15871/) (2025)
- [Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation](https://mlsanity.com/paper/2512.22745/) (2025)
- [3AM: 3egment Anything with Geometric Consistency in Videos](https://mlsanity.com/paper/2601.08831/) (2026)
- [EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation](https://mlsanity.com/paper/2606.08980/) (2026)
- [ClipGS: Clippable Gaussian Splatting for Interactive Cinematic Visualization of Volumetric Medical Data](https://mlsanity.com/paper/2507.06647/) (2025)
- [IGFuse: Interactive 3D Gaussian Scene Reconstruction via Multi-Scans Fusion](https://mlsanity.com/paper/2508.13153/) (2025)
- [GaussianTrimmer: Online Trimming Boundaries for 3DGS Segmentation](https://mlsanity.com/paper/2601.12683/) (2026)
- [B3-Seg: Camera-Free, Training-Free 3DGS Segmentation via Analytic EIG and Beta-Bernoulli Bayesian Updates](https://mlsanity.com/paper/2602.17134/) (2026)
- [DSG-World: Learning a 3D Gaussian World Model from Dual State Videos](https://mlsanity.com/paper/2506.05217/) (2025)
- [UniC-Lift: Unified 3D Instance Segmentation via Contrastive Learning](https://mlsanity.com/paper/2512.24763/) (2025)
- [Decoupling Motion and Geometry in 4D Gaussian Splatting](https://mlsanity.com/paper/2603.00952/) (2026)
- [Scene-Agnostic Object-Centric Representation Learning for 3D Gaussian Splatting](https://mlsanity.com/paper/2604.09045/) (2026)
- [Online Segment 3D Gaussians via Launching Virtual Drones](https://mlsanity.com/paper/2607.01628/) (2026)
- [CrashSplat: 2D to 3D Vehicle Damage Segmentation in Gaussian Splatting](https://mlsanity.com/paper/2509.23947/) (2025)
- [Learning to Sense for Driving: Joint Optics-Sensor-Model Co-Design for Semantic Segmentation](https://mlsanity.com/paper/2512.20815/) (2025)
- [G2P: Gaussian-to-Point Attribute Alignment for Boundary-Aware 3D Segmentation](https://mlsanity.com/paper/2601.03510/) (2026)
- [UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition](https://mlsanity.com/paper/2605.07356/) (2026)
- [BEA-GS: BEyond RAdiance Supervision in 3DGS for Precise Object Extraction](https://mlsanity.com/paper/2605.09662/) (2026)
- [Robust Prior-Guided Segmentation for Editable 3D Gaussian Splatting](https://mlsanity.com/paper/2605.16065/) (2026)
- [Geometry-Guided Modeling of Foundation Features Enables Generalizable Object Shape Deformation Learning](https://mlsanity.com/paper/2605.29661/) (2026)
- [SemDynReg: Semantics-Guided Deformation Regularization for Dynamic 3D Gaussian Splatting](https://mlsanity.com/paper/2606.28656/) (2026)
- [CNS-Edit++: Category-Agnostic 3D Editing with Coupled Neural Shape Representation](https://mlsanity.com/paper/2607.16577/) (2026)
- [LEGO: Leveled Language Gaussian Splatting](https://mlsanity.com/paper/2608.10057/) (2026)
- [Seed2GS: Camera-Free, Training-Free Object Extraction from 3D Gaussian Scenes via a Single Reference-View Grounding](https://mlsanity.com/paper/2608.11928/) (2026)
- [An overview of 3D Vision-Language Models](https://mlsanity.com/paper/2609.05583/) (2026)
- [PePESeg3D: Perception Prior Enhances Multi-Scale Segmentation for 3D Gaussian Splatting](https://mlsanity.com/paper/2609.28645/) (2026)
- [Every View Counts: View-Consistent Panoptic Quality for Multi-view Panoptic Segmentation](https://mlsanity.com/paper/2610.05911/) (2026)

## Abstract (arXiv)

Lifting multi-view 2D instance segmentation to a radiance field has proven to be effective to enhance 3D understanding. Existing methods rely on direct matching for end-to-end lifting, yielding inferior results; or employ a two-stage solution constrained by complex pre- or post-processing. In this work, we design a new end-to-end object-aware lifting approach, named Unified-Lift that provides accurate 3D segmentation based on the 3D Gaussian representation. To start, we augment each Gaussian point with an additional Gaussian-level feature learned using a contrastive loss to encode instance information. Importantly, we introduce a learnable object-level codebook to account for individual objects in the scene for an explicit object-level understanding and associate the encoded object-level features with the Gaussian-level point features for segmentation predictions. While promising, achieving effective codebook learning is non-trivial and a naive solution leads to degraded performance. Therefore, we formulate the association learning module and the noisy label filtering module for effective and robust codebook learning. We conduct experiments on three benchmarks: LERF-Masked, Replica, and Messy Rooms datasets. Both qualitative and quantitative results manifest that our Unified-Lift clearly outperforms existing methods in terms of segmentation quality and time efficiency. The code is publicly available at hrefhttps://github.com/Runsong123/Unified-Lifthttps://github.com/Runsong123/Unified-Lift.