mlsanityData to 14 Oct 2026

Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking

arXiv 2109.03805 ↗First posted 8 Sep 2021cs.CV cs.AI cs.LGcited by 166

Not on the map. Why?

Abstract (arXiv)

Panoptic scene understanding and tracking of dynamic agents are essential for robots and automated vehicles to navigate in urban environments. As LiDARs provide accurate illumination-independent geometric depictions of the scene, performing these tasks using LiDAR point clouds provides reliable predictions. However, existing datasets lack diversity in the type of urban scenes and have a limited number of dynamic object instances which hinders both learning of these tasks as well as credible benchmarking of the developed methods. In this paper, we introduce the large-scale Panoptic nuScenes benchmark dataset that extends our popular nuScenes dataset with point-wise groundtruth annotations for semantic segmentation, panoptic segmentation, and panoptic tracking tasks. To facilitate comparison, we provide several strong baselines for each of these tasks on our proposed dataset. Moreover, we analyze the drawbacks of the existing metrics for panoptic tracking and propose the novel instance-centric PAT metric that addresses the concerns. We present exhaustive experiments that demonstrate the utility of Panoptic nuScenes compared to existing datasets and make the online evaluation server available at nuScenes.org. We believe that this extension will accelerate the research of novel methods for scene understanding of dynamic urban environments.

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 8 Sep 2021.

Builds on

28 of 36 references.

References · 28

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  2. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  3. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  4. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  5. 2018Panoptic Segmentationcited by 802
  6. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  7. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 585
  8. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  9. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 389
  10. 2020PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentationcited by 275
  11. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 240
  12. 2020A2D2: Audi Autonomous Driving Datasetcited by 190
16 more
  1. 2021(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Networkcited by 164
  2. 2019Seamless Scene Segmentationcited by 112
  3. 2020EfficientPS: Efficient Panoptic Segmentationcited by 111
  4. 2021Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentationcited by 107
  5. 2020AMVNet: Assertion-based Multi-View Fusion Network for LiDAR Semantic Segmentationcited by 87
  6. 2020LiDAR-based Panoptic Segmentation via Dynamic Shifting Networkcited by 84
  7. 2020Toronto-3D: A Large-scale Mobile LiDAR Dataset for Semantic Segmentation of Urban Roadwayscited by 77
  8. 2021EfficientLPS: Efficient LiDAR Panoptic Segmentationcited by 76
  9. 2021STEP: Segmenting and Tracking Every Pixelcited by 73
  10. 2021LCDNet: Deep Loop Closure Detection and Point Cloud Registration for LiDAR SLAMcited by 67
  11. 20214D Panoptic LiDAR Segmentationcited by 65
  12. 2020Probabilistic 3D Multi-Object Tracking for Autonomous Drivingcited by 64
  13. 2020MOPT: Multi-Object Panoptic Trackingcited by 53
  14. 2020Panoster: End-to-end Panoptic Segmentation of LiDAR Point Cloudscited by 46
  15. 2020A Benchmark for LiDAR-based Panoptic Segmentation based on KITTIcited by 40
  16. 2021Learnable Online Graph Representations for 3D Multi-Object Trackingcited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 16; 2023: 56; 2024: 35; 2025: 33; 2026: 21.

166 papers.

Sort:

By area:

  1. 2023Point Transformer V3: Simpler, Faster, Strongercited by 589
  2. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
  3. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 332
  4. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 293
  5. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 221
  6. 2023Scene as Occupancycited by 209
  7. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  8. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  9. 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 129
  10. 2023Rethinking Range View Representation for LiDAR Segmentationcited by 128
  11. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 125
  12. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 115
38 more
  1. 2022LaserMix for Semi-Supervised LiDAR Semantic Segmentationcited by 93
  2. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  3. 2023SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Drivingcited by 75
  4. 2022PolarMix: A General Data Augmentation Technique for LiDAR Point Cloudscited by 74
  5. 2022LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perceptioncited by 74
  6. 2024Towards Realistic Scene Generation with LiDAR Diffusion Modelscited by 69
  7. 2023MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Drivingcited by 68
  8. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
  9. 20253D and 4D World Modeling: A Surveycited by 67
  10. 2024The Neglected Tails in Vision-Language Modelscited by 63
  11. 2022Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmapcited by 59
  12. 2023UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebasecited by 57
  13. 2023Towards Label-free Scene Understanding by Vision Foundation Modelscited by 54
  14. 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 54
  15. 2023FRNet: Frustum-Range Networks for Scalable LiDAR Segmentationcited by 50
  16. 2024Better Call SAL: Towards Learning to Segment Anything in Lidarcited by 47
  17. 20233D Semantic Segmentation in the Wild: Learning Generalized Models for Adverse-Condition Point Cloudscited by 46
  18. 2021ConDA: Unsupervised Domain Adaptation for LiDAR Segmentation via Regularized Domain Concatenationcited by 41
  19. 2024Scaling Diffusion Models to Real-World 3D LiDAR Scene Completioncited by 40
  20. 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
  21. 2024ViewFormer: Exploring Spatiotemporal Modeling for Multi-View 3D Occupancy Perception via View-Guided Transformerscited by 37
  22. 20224D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregationcited by 35
  23. 20244D Contrastive Superflows are Dense 3D Representation Learnerscited by 33
  24. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  25. 20223D Multi-Object Tracking Using Graph Neural Networks with Cross-Edge Modality Attentioncited by 27
  26. 2024Is Your LiDAR Placement Optimized for 3D Scene Understanding?cited by 26
  27. 2022Domain generalization of 3D semantic segmentation in autonomous drivingcited by 25
  28. 2024Multi-Space Alignments Towards Universal LiDAR Segmentationcited by 25
  29. 2023Mask4Former: Mask Transformer for 4D Panoptic Segmentationcited by 24
  30. 2025GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flowcited by 24
  31. 2024An End-to-End Robust Point Cloud Semantic Segmentation Network with Single-Step Conditional Diffusion Modelscited by 23
  32. 2024OccFlowNet: Towards Self-supervised Occupancy Estimation via Differentiable Rendering and Occupancy Flowcited by 22
  33. 2023SALUDA: Surface-based Automotive Lidar Unsupervised Domain Adaptationcited by 21
  34. 2023Learning and Aggregating Lane Graphs for Urban Automated Drivingcited by 20
  35. 2023Learning to Adapt SAM for Segmenting Cross-domain Point Cloudscited by 20
  36. 2021Unsupervised Domain Adaptation for LiDAR Panoptic Segmentationcited by 19
  37. 20234D Panoptic Segmentation as Invariant and Equivariant Field Predictioncited by 19
  38. 2025LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenescited by 19

50 most cited of 166 All 166 (JSON)