# LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes

arXiv: https://arxiv.org/abs/2405.00900 · HTML: https://mlsanity.com/paper/2405.00900/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 50 references that are in the map.
- First posted on arXiv 1 May 2024.

## Builds on (most cited first, 30 of 41)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis](https://mlsanity.com/paper/2303.16196/) (2023)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving](https://mlsanity.com/paper/2112.12610/) (2021)
- [StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Views](https://mlsanity.com/paper/2306.04988/) (2023)
- [S-NeRF: Neural Radiance Fields for Street Views](https://mlsanity.com/paper/2303.00749/) (2023)
- [Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes](https://mlsanity.com/paper/2304.03266/) (2023)
- [Behind the Scenes: Density Fields for Single View Reconstruction](https://mlsanity.com/paper/2301.07668/) (2023)
- [Neural Point Light Fields](https://mlsanity.com/paper/2112.01473/) (2021)

## Built on by (most cited first, 14 of 14)

- [DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation](https://mlsanity.com/paper/2410.13571/) (2024)
- [StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models](https://mlsanity.com/paper/2412.13188/) (2024)
- [CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Driving](https://mlsanity.com/paper/2503.06744/) (2025)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting](https://mlsanity.com/paper/2507.18678/) (2025)
- [Street Gaussians without 3D Object Tracker](https://mlsanity.com/paper/2412.05548/) (2024)
- [FlexDrive: Toward Trajectory Flexibility in Driving Scene Reconstruction and Rendering](https://mlsanity.com/paper/2502.21093/) (2025)
- [LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agents](https://mlsanity.com/paper/2512.17445/) (2025)
- [HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles](https://mlsanity.com/paper/2602.21333/) (2026)
- [AutoScape: Geometry-Consistent Long-Horizon Scene Generation](https://mlsanity.com/paper/2510.20726/) (2025)
- [HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes](https://mlsanity.com/paper/2604.04887/) (2026)
- [FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gain](https://mlsanity.com/paper/2511.21113/) (2025)
- [Self-Supervised Multimodal NeRF for Autonomous Driving](https://mlsanity.com/paper/2506.19615/) (2025)
- [GARO: Geometry-Aware Redundancy Optimization for Real-Time and High-Fidelity Dynamic Gaussian Splatting](https://mlsanity.com/paper/2609.23509/) (2026)

## Abstract (arXiv)

Photorealistic simulation plays a crucial role in applications such as autonomous driving, where advances in neural radiance fields (NeRFs) may allow better scalability through the automatic creation of digital 3D assets. However, reconstruction quality suffers on street scenes due to largely collinear camera motions and sparser samplings at higher speeds. On the other hand, the application often demands rendering from camera views that deviate from the inputs to accurately simulate behaviors like lane changes. In this paper, we propose several insights that allow a better utilization of Lidar data to improve NeRF quality on street scenes. First, our framework learns a geometric scene representation from Lidar, which is fused with the implicit grid-based representation for radiance decoding, thereby supplying stronger geometric information offered by explicit point cloud. Second, we put forth a robust occlusion-aware depth supervision scheme, which allows utilizing densified Lidar points by accumulation. Third, we generate augmented training views from Lidar points for further improvement. Our insights translate to largely improved novel view synthesis under real driving scenes.