# Quantifying the LiDAR Sim-to-Real Domain Shift: A Detailed Investigation Using Object Detectors and Analyzing Point Clouds at Target-Level

arXiv: https://arxiv.org/abs/2303.01899 · HTML: https://mlsanity.com/paper/2303.01899/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 3 Mar 2023.

## Builds on (most cited first, 28 of 28)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data](https://mlsanity.com/paper/1909.00889/) (2019)
- [Disentangling Monocular 3D Object Detection](https://mlsanity.com/paper/1905.12365/) (2019)
- [Train in Germany, Test in The USA: Making 3D Object Detectors Generalize](https://mlsanity.com/paper/2005.08139/) (2020)
- [LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World](https://mlsanity.com/paper/2006.09348/) (2020)
- [Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes](https://mlsanity.com/paper/1708.01566/) (2017)
- [Understanding How Image Quality Affects Deep Neural Networks](https://mlsanity.com/paper/1604.04004/) (2016)
- [Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data](https://mlsanity.com/paper/1810.10093/) (2018)
- [A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving](https://mlsanity.com/paper/1804.00103/) (2018)
- [A Survey on Deep Domain Adaptation for LiDAR Perception](https://mlsanity.com/paper/2106.02377/) (2021)
- [Augmented LiDAR Simulator for Autonomous Driving](https://mlsanity.com/paper/1811.07112/) (2018)
- [How much real data do we actually need: Analyzing object detection performance using synthetic and real data](https://mlsanity.com/paper/1907.07061/) (2019)
- [See Eye to Eye: A Lidar-Agnostic 3D Detection Framework for Unsupervised Multi-Target Domain Adaptation](https://mlsanity.com/paper/2111.09450/) (2021)
- [Mixing Real and Synthetic Data to Enhance Neural Network Training -- A Review of Current Approaches](https://mlsanity.com/paper/2007.08781/) (2020)
- [A Realism Metric for Generated LiDAR Point Clouds](https://mlsanity.com/paper/2208.14958/) (2022)
- [Reducing the Amount of Real World Data for Object Detector Training with Synthetic Data](https://mlsanity.com/paper/2202.00632/) (2022)
- [Low Cost, Open-Source Testbed to Enable Full-Sized Automated Vehicle Research](https://mlsanity.com/paper/1708.07771/) (2017)

## Built on by (most cited first, 7 of 7)

- [GeoAdapt: Self-Supervised Test-Time Adaptation in LiDAR Place Recognition Using Geometric Priors](https://mlsanity.com/paper/2308.04638/) (2023)
- [Vulnerable Road User Detection and Safety Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2405.19202/) (2024)
- [Synth It Like KITTI: Synthetic Data Generation for Object Detection in Driving Scenarios](https://mlsanity.com/paper/2502.15076/) (2025)
- [TrueCity: Real and Simulated Urban Data for Cross-Domain 3D Scene Understanding](https://mlsanity.com/paper/2511.07007/) (2025)
- [DeepSTEP -- Deep Learning-Based Spatio-Temporal End-To-End Perception for Autonomous Vehicles](https://mlsanity.com/paper/2305.06820/) (2023)
- [Mind the Domain Gap: Measuring the Domain Gap Between Real-World and Synthetic Point Clouds for Automated Driving Development](https://mlsanity.com/paper/2505.17959/) (2025)
- [End-to-End Crop Row Navigation via LiDAR-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2509.18608/) (2025)

## Abstract (arXiv)

LiDAR object detection algorithms based on neural networks for autonomous driving require large amounts of data for training, validation, and testing. As real-world data collection and labeling are time-consuming and expensive, simulation-based synthetic data generation is a viable alternative. However, using simulated data for the training of neural networks leads to a domain shift of training and testing data due to differences in scenes, scenarios, and distributions. In this work, we quantify the sim-to-real domain shift by means of LiDAR object detectors trained with a new scenario-identical real-world and simulated dataset. In addition, we answer the questions of how well the simulated data resembles the real-world data and how well object detectors trained on simulated data perform on real-world data. Further, we analyze point clouds at the target-level by comparing real-world and simulated point clouds within the 3D bounding boxes of the targets. Our experiments show that a significant sim-to-real domain shift exists even for our scenario-identical datasets. This domain shift amounts to an average precision reduction of around 14 % for object detectors trained with simulated data. Additional experiments reveal that this domain shift can be lowered by introducing a simple noise model in simulation. We further show that a simple downsampling method to model real-world physics does not influence the performance of the object detectors.