# A Lightweight Machine Learning Pipeline for LiDAR-simulation

arXiv: https://arxiv.org/abs/2208.03130 · HTML: https://mlsanity.com/paper/2208.03130/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 5 Aug 2022.

## Builds on (most cited first, 18 of 18)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World](https://mlsanity.com/paper/2006.09348/) (2020)
- [STEP: Segmenting and Tracking Every Pixel](https://mlsanity.com/paper/2102.11859/) (2021)

## Built on by (most cited first, 3 of 3)

- [From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Driving](https://mlsanity.com/paper/2305.13960/) (2023)
- [GAN-Based LiDAR Intensity Simulation](https://mlsanity.com/paper/2311.15415/) (2023)
- [Toward Physics-Aware Deep Learning Architectures for LiDAR Intensity Simulation](https://mlsanity.com/paper/2404.15774/) (2024)

## Abstract (arXiv)

Virtual testing is a crucial task to ensure safety in autonomous driving, and sensor simulation is an important task in this domain. Most current LiDAR simulations are very simplistic and are mainly used to perform initial tests, while the majority of insights are gathered on the road. In this paper, we propose a lightweight approach for more realistic LiDAR simulation that learns a real sensor's behavior from test drive data and transforms this to the virtual domain. The central idea is to cast the simulation into an image-to-image translation problem. We train our pix2pix based architecture on two real world data sets, namely the popular KITTI data set and the Audi Autonomous Driving Dataset which provide both, RGB and LiDAR images. We apply this network on synthetic renderings and show that it generalizes sufficiently from real images to simulated images. This strategy enables to skip the sensor-specific, expensive and complex LiDAR physics simulation in our synthetic world and avoids oversimplification and a large domain-gap through the clean synthetic environment.