mlsanityData to 7 Oct 2026

LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World

arXiv 2006.09348 ↗First posted 16 Jun 2020cs.CVcited by 137

Not on the map. Why?

Abstract (arXiv)

We tackle the problem of producing realistic simulations of LiDAR point clouds, the sensor of preference for most self-driving vehicles. We argue that, by leveraging real data, we can simulate the complex world more realistically compared to employing virtual worlds built from CAD/procedural models. Towards this goal, we first build a large catalog of 3D static maps and 3D dynamic objects by driving around several cities with our self-driving fleet. We can then generate scenarios by selecting a scene from our catalog and "virtually" placing the self-driving vehicle (SDV) and a set of dynamic objects from the catalog in plausible locations in the scene. To produce realistic simulations, we develop a novel simulator that captures both the power of physics-based and learning-based simulation. We first utilize ray casting over the 3D scene and then use a deep neural network to produce deviations from the physics-based simulation, producing realistic LiDAR point clouds. We showcase LiDARsim's usefulness for perception algorithms-testing on long-tail events and end-to-end closed-loop evaluation on safety-critical scenarios.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 15; 2023: 30; 2024: 31; 2025: 24; 2026: 13.

137 papers.

Sort:

By area:

  1. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  2. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  3. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  4. 2021OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communicationcited by 304
  5. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  6. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splattingcited by 218
  7. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  8. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  9. 2022V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Drivingcited by 157
  10. 2023V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perceptioncited by 155
  11. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 132
  12. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
38 more
  1. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  2. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  3. 2021LookOut: Diverse Multi-Future Prediction and Planning for Self-Drivingcited by 96
  4. 2022Learning to Generate Realistic LiDAR Point Cloudscited by 91
  5. 2023The Waymo Open Sim Agents Challengecited by 86
  6. 2021SceneGen: Learning to Generate Realistic Traffic Scenescited by 83
  7. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
  8. 2023UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generationcited by 76
  9. 2021GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Drivingcited by 75
  10. 2024Towards Realistic Scene Generation with LiDAR Diffusion Modelscited by 67
  11. 2023Neural LiDAR Fields for Novel View Synthesiscited by 65
  12. 2021Multi-Modal 3D Object Detection in Autonomous Driving: a Surveycited by 63
  13. 2025A Survey of World Models for Autonomous Drivingcited by 60
  14. 2022Motion Inspired Unsupervised Perception and Prediction in Autonomous Drivingcited by 59
  15. 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
  16. 2021Adversarial Attacks On Multi-Agent Communicationcited by 48
  17. 2023LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fieldscited by 46
  18. 2021Keypoints-Based Deep Feature Fusion for Cooperative Vehicle Detection of Autonomous Drivingcited by 45
  19. 2024RangeLDM: Fast Realistic LiDAR Point Cloud Generationcited by 45
  20. 2023NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fieldscited by 44
  21. 2023Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perceptioncited by 38
  22. 2023Rethinking Closed-loop Training for Autonomous Drivingcited by 29
  23. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  24. 2024S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generationcited by 28
  25. 2021Learning to Drop Points for LiDAR Scan Synthesiscited by 27
  26. 2023CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulationcited by 26
  27. 2024LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesiscited by 26
  28. 2024Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Videocited by 26
  29. 2024Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformercited by 26
  30. 2021Model-free Vehicle Tracking and State Estimation in Point Cloud Sequencescited by 25
  31. 2021Fooling LiDAR Perception via Adversarial Trajectory Perturbationcited by 25
  32. 2023GINA-3D: Learning to Generate Implicit Neural Assets in the Wildcited by 25
  33. 2024Is Your LiDAR Placement Optimized for 3D Scene Understanding?cited by 25
  34. 2022SoK: On the Semantic AI Security in Autonomous Drivingcited by 22
  35. 2023Dynamic LiDAR Re-simulation using Compositional Neural Fieldscited by 22
  36. 2022A Survey and Framework of Cooperative Perception: From Heterogeneous Singleton to Hierarchical Cooperationcited by 21
  37. 2023Reconstructing Objects in-the-wild for Realistic Sensor Simulationcited by 21
  38. 2022PCGen: Point Cloud Generator for LiDAR Simulationcited by 20

50 most cited of 137 All 137 (JSON)