# PLGS: Robust Panoptic Lifting with 3D Gaussian Splatting

arXiv: https://arxiv.org/abs/2410.17505 · HTML: https://mlsanity.com/paper/2410.17505/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 74 references that are in the map.
- First posted on arXiv 23 Oct 2024.

## Builds on (most cited first, 30 of 58)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering](https://mlsanity.com/paper/2312.00109/) (2023)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)

## Built on by (most cited first, 17 of 17)

- [PanSt3R: Multi-view Consistent Panoptic Segmentation](https://mlsanity.com/paper/2506.21348/) (2025)
- [Occam's LGS: An Efficient Approach for Language Gaussian Splatting](https://mlsanity.com/paper/2412.01807/) (2024)
- [PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understanding](https://mlsanity.com/paper/2503.18107/) (2025)
- [GazeGaussian: High-Fidelity Gaze Redirection with 3D Gaussian Splatting](https://mlsanity.com/paper/2411.12981/) (2024)
- [Pointmap Association and Piecewise-Plane Constraint for Consistent and Compact 3D Gaussian Segmentation Field](https://mlsanity.com/paper/2502.16303/) (2025)
- [ParkGaussian: Surround-view 3D Gaussian Splatting for Autonomous Parking](https://mlsanity.com/paper/2601.01386/) (2026)
- [OmniIndoor3D: Comprehensive Indoor 3D Reconstruction](https://mlsanity.com/paper/2505.20610/) (2025)
- [DNRSelect: Active Best View Selection for Deferred Neural Rendering](https://mlsanity.com/paper/2501.12150/) (2025)
- [PanopticSplatting: End-to-End Panoptic Gaussian Splatting](https://mlsanity.com/paper/2503.18073/) (2025)
- [DynSUP: Dynamic Gaussian Splatting from An Unposed Image Pair](https://mlsanity.com/paper/2412.00851/) (2024)
- [Queryable 3D Scene Representation: A Multi-Modal Framework for Semantic Reasoning and Robotic Task Planning](https://mlsanity.com/paper/2509.20077/) (2025)
- [FrontierGS: Progressive View-Space Frontier Expansion for Sparse 3D Gaussian Splatting](https://mlsanity.com/paper/2511.16030/) (2025)
- [Toward Fine-Grained Facial Control in 3D Talking Head Generation](https://mlsanity.com/paper/2602.09736/) (2026)
- [GraspFoM: Towards Reconstruction-Driven Robotic Grasping with 3D Foundation Priors](https://mlsanity.com/paper/2606.08440/) (2026)
- [Accurate and Complete Surface Reconstruction from 3D Gaussians via Direct SDF Learning](https://mlsanity.com/paper/2509.07493/) (2025)
- [Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation](https://mlsanity.com/paper/2607.19765/) (2026)
- [Compressing 3D Gaussian Splatting via Cross-Representation Priors](https://mlsanity.com/paper/2609.23005/) (2026)

## Abstract (arXiv)

Previous methods utilize the Neural Radiance Field (NeRF) for panoptic lifting, while their training and rendering speed are unsatisfactory. In contrast, 3D Gaussian Splatting (3DGS) has emerged as a prominent technique due to its rapid training and rendering speed. However, unlike NeRF, the conventional 3DGS may not satisfy the basic smoothness assumption as it does not rely on any parameterized structures to render (e.g., MLPs). Consequently, the conventional 3DGS is, in nature, more susceptible to noisy 2D mask supervision. In this paper, we propose a new method called PLGS that enables 3DGS to generate consistent panoptic segmentation masks from noisy 2D segmentation masks while maintaining superior efficiency compared to NeRF-based methods. Specifically, we build a panoptic-aware structured 3D Gaussian model to introduce smoothness and design effective noise reduction strategies. For the semantic field, instead of initialization with structure from motion, we construct reliable semantic anchor points to initialize the 3D Gaussians. We then use these anchor points as smooth regularization during training. Additionally, we present a self-training approach using pseudo labels generated by merging the rendered masks with the noisy masks to enhance the robustness of PLGS. For the instance field, we project the 2D instance masks into 3D space and match them with oriented bounding boxes to generate cross-view consistent instance masks for supervision. Experiments on various benchmarks demonstrate that our method outperforms previous state-of-the-art methods in terms of both segmentation quality and speed.