# PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving

arXiv: https://arxiv.org/abs/2112.12610 · HTML: https://mlsanity.com/paper/2112.12610/

## Key facts

- Cited by 190 papers in the map (data through 14 Oct 2026).
- Placed in World modeling from LiDAR generation, one of 11 directions in World models.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 23 Dec 2021.

## Placed in

- World models / [World modeling from LiDAR generation](https://mlsanity.com/wm/world-modeling-from-lidar-generation/)

## Builds on (most cited first, 17 of 17)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [Fast Point R-CNN](https://mlsanity.com/paper/1908.02990/) (2019)
- [A*3D Dataset: Towards Autonomous Driving in Challenging Environments](https://mlsanity.com/paper/1909.07541/) (2019)
- [Labels Are Not Perfect: Inferring Spatial Uncertainty in Object Detection](https://mlsanity.com/paper/2012.12195/) (2020)
- [Cirrus: A Long-range Bi-pattern LiDAR Dataset](https://mlsanity.com/paper/2012.02938/) (2020)

## Built on by (most cited first, 30 of 190)

- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Panacea: Panoramic and Controllable Video Generation for Autonomous Driving](https://mlsanity.com/paper/2311.16813/) (2023)
- [OmniRe: Omni Urban Scene Reconstruction](https://mlsanity.com/paper/2408.16760/) (2024)
- [NeuRAD: Neural Rendering for Autonomous Driving](https://mlsanity.com/paper/2311.15260/) (2023)
- [MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds](https://mlsanity.com/paper/2412.06974/) (2024)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation](https://mlsanity.com/paper/2410.13571/) (2024)
- [Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe](https://mlsanity.com/paper/2209.05324/) (2022)
- [HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving](https://mlsanity.com/paper/2412.01718/) (2024)
- [ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration](https://mlsanity.com/paper/2411.19548/) (2024)
- [Zenseact Open Dataset: A large-scale and diverse multimodal dataset for autonomous driving](https://mlsanity.com/paper/2305.02008/) (2023)
- [UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation](https://mlsanity.com/paper/2311.01448/) (2023)
- [Multi-modal Sensor Fusion for Auto Driving Perception: A Survey](https://mlsanity.com/paper/2202.02703/) (2022)
- [Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys](https://mlsanity.com/paper/2303.17220/) (2023)
- [SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Driving](https://mlsanity.com/paper/2306.09001/) (2023)
- [SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving](https://mlsanity.com/paper/2411.16816/) (2024)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving](https://mlsanity.com/paper/2404.07762/) (2024)
- [Uncertainty-aware Vision-based Metric Cross-view Geolocalization](https://mlsanity.com/paper/2211.12145/) (2022)
- [AutoSplat: Constrained Gaussian Splatting for Autonomous Driving Scene Reconstruction](https://mlsanity.com/paper/2407.02598/) (2024)
- [StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models](https://mlsanity.com/paper/2412.13188/) (2024)
- [Three Pillars improving Vision Foundation Model Distillation for Lidar](https://mlsanity.com/paper/2310.17504/) (2023)
- [Driving on Registers](https://mlsanity.com/paper/2601.05083/) (2026)
- [3DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Views](https://mlsanity.com/paper/2406.04875/) (2024)
- [Real-Time Neural Rasterization for Large Scenes](https://mlsanity.com/paper/2311.05607/) (2023)
- [ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representation](https://mlsanity.com/paper/2503.18438/) (2025)

## Abstract (arXiv)

The accelerating development of autonomous driving technology has placed greater demands on obtaining large amounts of high-quality data. Representative, labeled, real world data serves as the fuel for training deep learning networks, critical for improving self-driving perception algorithms. In this paper, we introduce PandaSet, the first dataset produced by a complete, high-precision autonomous vehicle sensor kit with a no-cost commercial license. The dataset was collected using one 360deg mechanical spinning LiDAR, one forward-facing, long-range LiDAR, and 6 cameras. The dataset contains more than 100 scenes, each of which is 8 seconds long, and provides 28 types of labels for object classification and 37 types of labels for semantic segmentation. We provide baselines for LiDAR-only 3D object detection, LiDAR-camera fusion 3D object detection and LiDAR point cloud segmentation. For more details about PandaSet and the development kit, see https://scale.com/open-datasets/pandaset.