# PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness

arXiv: https://arxiv.org/abs/2312.02158 · HTML: https://mlsanity.com/paper/2312.02158/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 79 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Builds on (most cited first, 30 of 63)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion](https://mlsanity.com/paper/2012.03762/) (2020)
- [LMSCNet: Lightweight Multiscale 3D Semantic Completion](https://mlsanity.com/paper/2008.10559/) (2020)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Robo3D: Towards Robust and Reliable 3D Perception against Corruptions](https://mlsanity.com/paper/2303.17597/) (2023)

## Built on by (most cited first, 30 of 42)

- [A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective](https://mlsanity.com/paper/2405.05173/) (2024)
- [4D Contrastive Superflows are Dense 3D Representation Learners](https://mlsanity.com/paper/2407.06190/) (2024)
- [Is Your LiDAR Placement Optimized for 3D Scene Understanding?](https://mlsanity.com/paper/2403.17009/) (2024)
- [LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera](https://mlsanity.com/paper/2407.16197/) (2024)
- [ProtoOcc: Accurate, Efficient 3D Occupancy Prediction Using Dual Branch Encoder-Prototype Query Decoder](https://mlsanity.com/paper/2412.08774/) (2024)
- [GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention](https://mlsanity.com/paper/2505.10685/) (2025)
- [HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction](https://mlsanity.com/paper/2408.09104/) (2024)
- [VoxDet: Rethinking 3D Semantic Occupancy Prediction as Dense Object Detection](https://mlsanity.com/paper/2506.04623/) (2025)
- [An Empirical Study of Training State-of-the-Art LiDAR Segmentation Models](https://mlsanity.com/paper/2405.14870/) (2024)
- [Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding](https://mlsanity.com/paper/2403.17010/) (2024)
- [OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation](https://mlsanity.com/paper/2412.11183/) (2024)
- [Occupancy Learning with Spatiotemporal Memory](https://mlsanity.com/paper/2508.04705/) (2025)
- [L2COcc: Lightweight Camera-Centric Semantic Scene Completion via Distillation of LiDAR Model](https://mlsanity.com/paper/2503.12369/) (2025)
- [How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation](https://mlsanity.com/paper/2505.18956/) (2025)
- [Towards Learning to Complete Anything in Lidar](https://mlsanity.com/paper/2504.12264/) (2025)
- [OCCUQ: Exploring Efficient Uncertainty Quantification for 3D Occupancy Prediction](https://mlsanity.com/paper/2503.10605/) (2025)
- [OccAny: Generalized Unconstrained Urban 3D Occupancy](https://mlsanity.com/paper/2603.23502/) (2026)
- [Delta-Triplane Transformers as Occupancy World Models](https://mlsanity.com/paper/2503.07338/) (2025)
- [DOC-Depth: A novel approach for dense depth ground truth generation](https://mlsanity.com/paper/2502.02144/) (2025)
- [TrackOcc: Camera-based 4D Panoptic Occupancy Tracking](https://mlsanity.com/paper/2503.08471/) (2025)
- [Uncertainty-Instructed Structure Injection for Generalizable HD Map Construction](https://mlsanity.com/paper/2503.23109/) (2025)
- [Audio-Driven Talking Face Video Generation with Joint Uncertainty Learning](https://mlsanity.com/paper/2504.18810/) (2025)
- [IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposals](https://mlsanity.com/paper/2506.20671/) (2025)
- [Enhancing 3D Semantic Scene Completion with a Refinement Module](https://mlsanity.com/paper/2512.18363/) (2025)
- [Latent Gaussian Splatting for 4D Panoptic Occupancy Tracking](https://mlsanity.com/paper/2602.23172/) (2026)
- [Out-of-Distribution Semantic Occupancy Prediction](https://mlsanity.com/paper/2506.21185/) (2025)
- [TGSFormer: Scalable Temporal Gaussian Splatting for Embodied Semantic Scene Completion](https://mlsanity.com/paper/2512.00300/) (2025)
- [The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models](https://mlsanity.com/paper/2601.08876/) (2026)
- [Point-based Instance Completion with Scene Constraints](https://mlsanity.com/paper/2504.05698/) (2025)
- [An Instance-Centric Panoptic Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2603.27238/) (2026)

## Abstract (arXiv)

We propose the task of Panoptic Scene Completion (PSC) which extends the recently popular Semantic Scene Completion (SSC) task with instance-level information to produce a richer understanding of the 3D scene. Our PSC proposal utilizes a hybrid mask-based technique on the non-empty voxels from sparse multi-scale completions. Whereas the SSC literature overlooks uncertainty which is critical for robotics applications, we instead propose an efficient ensembling to estimate both voxel-wise and instance-wise uncertainties along PSC. This is achieved by building on a multi-input multi-output (MIMO) strategy, while improving performance and yielding better uncertainty for little additional compute. Additionally, we introduce a technique to aggregate permutation-invariant mask predictions. Our experiments demonstrate that our method surpasses all baselines in both Panoptic Scene Completion and uncertainty estimation on three large-scale autonomous driving datasets. Our code and data are available at https://astra-vision.github.io/PaSCo .