# NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields

arXiv: https://arxiv.org/abs/2304.14811 · HTML: https://mlsanity.com/paper/2304.14811/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 51 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 30 of 40)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [Dense Depth Priors for Neural Radiance Fields from Sparse Input Views](https://mlsanity.com/paper/2112.03288/) (2021)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks](https://mlsanity.com/paper/2103.03231/) (2021)
- [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021)
- [NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo](https://mlsanity.com/paper/2109.01129/) (2021)
- [IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments](https://mlsanity.com/paper/1811.10200/) (2018)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)
- [LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World](https://mlsanity.com/paper/2006.09348/) (2020)

## Built on by (most cited first, 30 of 44)

- [UniPAD: A Universal Pre-training Paradigm for Autonomous Driving](https://mlsanity.com/paper/2310.08370/) (2023)
- [LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis](https://mlsanity.com/paper/2404.02742/) (2024)
- [Dynamic LiDAR Re-simulation using Compositional Neural Fields](https://mlsanity.com/paper/2312.05247/) (2023)
- [Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities](https://mlsanity.com/paper/2401.08045/) (2024)
- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [Neural Radiance Field in Autonomous Driving: A Survey](https://mlsanity.com/paper/2404.13816/) (2024)
- [GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control](https://mlsanity.com/paper/2505.22421/) (2025)
- [AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis](https://mlsanity.com/paper/2402.17483/) (2024)
- [Applications of Large Scale Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2311.12144/) (2023)
- [Neural Radiance Field-based Visual Rendering: A Comprehensive Review](https://mlsanity.com/paper/2404.00714/) (2024)
- [LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation](https://mlsanity.com/paper/2412.15199/) (2024)
- [Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB](https://mlsanity.com/paper/2411.19474/) (2024)
- [RealEngine: Simulating Autonomous Driving in Realistic Context](https://mlsanity.com/paper/2505.16902/) (2025)
- [Bayesian NeRF: Quantifying Uncertainty with Volume Density for Neural Implicit Fields](https://mlsanity.com/paper/2404.06727/) (2024)
- [LOGen: Toward Lidar Object Generation by Point Diffusion](https://mlsanity.com/paper/2412.07385/) (2024)
- [Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method](https://mlsanity.com/paper/2510.22973/) (2025)
- [GeoNLF: Geometry guided Pose-Free Neural LiDAR Fields](https://mlsanity.com/paper/2407.05597/) (2024)
- [A Survey on 3D Reconstruction Techniques in Plant Phenotyping: From Classical Methods to Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and Beyond](https://mlsanity.com/paper/2505.00737/) (2025)
- [PC-NeRF: Parent-Child Neural Radiance Fields Using Sparse LiDAR Frames in Autonomous Driving Environments](https://mlsanity.com/paper/2402.09325/) (2024)
- [Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene](https://mlsanity.com/paper/2502.06682/) (2025)
- [LiDAR-EDIT: LiDAR Data Generation by Editing the Object Layouts in Real-World Scenes](https://mlsanity.com/paper/2412.00592/) (2024)
- [SimULi: Real-Time LiDAR and Camera Simulation with Unscented Transforms](https://mlsanity.com/paper/2510.12901/) (2025)
- [PC-NeRF: Parent-Child Neural Radiance Fields under Partial Sensor Data Loss in Autonomous Driving Environments](https://mlsanity.com/paper/2310.00874/) (2023)
- [Gated Fields: Learning Scene Reconstruction from Gated Videos](https://mlsanity.com/paper/2405.19819/) (2024)
- [GLiDR: Topologically Regularized Graph Generative Network for Sparse LiDAR Point Clouds](https://mlsanity.com/paper/2312.00068/) (2023)
- [PillarGen: Enhancing Radar Point Cloud Density and Quality via Pillar-based Point Generation Network](https://mlsanity.com/paper/2403.01663/) (2024)
- [SN-LiDAR: Semantic Neural Fields for Novel Space-time View LiDAR Synthesis](https://mlsanity.com/paper/2504.08361/) (2025)
- [MOVES: Movable and Moving LiDAR Scene Segmentation in Label-Free settings using Static Reconstruction](https://mlsanity.com/paper/2306.14812/) (2023)
- [TeX-NeRF: Neural Radiance Fields from Pseudo-TeX Vision](https://mlsanity.com/paper/2410.04873/) (2024)
- [Simultaneous Diffusion Sampling for Conditional LiDAR Generation](https://mlsanity.com/paper/2410.11628/) (2024)

## Abstract (arXiv)

Labeling LiDAR point clouds for training autonomous driving is extremely expensive and difficult. LiDAR simulation aims at generating realistic LiDAR data with labels for training and verifying self-driving algorithms more efficiently. Recently, Neural Radiance Fields (NeRF) have been proposed for novel view synthesis using implicit reconstruction of 3D scenes. Inspired by this, we present NeRF-LIDAR, a novel LiDAR simulation method that leverages real-world information to generate realistic LIDAR point clouds. Different from existing LiDAR simulators, we use real images and point cloud data collected by self-driving cars to learn the 3D scene representation, point cloud generation and label rendering. We verify the effectiveness of our NeRF-LiDAR by training different 3D segmentation models on the generated LiDAR point clouds. It reveals that the trained models are able to achieve similar accuracy when compared with the same model trained on the real LiDAR data. Besides, the generated data is capable of boosting the accuracy through pre-training which helps reduce the requirements of the real labeled data.