# SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving

arXiv: https://arxiv.org/abs/2411.16816 · HTML: https://mlsanity.com/paper/2411.16816/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 52 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 30 of 43)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [Mip-Splatting: Alias-free 3D Gaussian Splatting](https://mlsanity.com/paper/2311.16493/) (2023)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes](https://mlsanity.com/paper/2312.07920/) (2023)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [ReconFusion: 3D Reconstruction with Diffusion Priors](https://mlsanity.com/paper/2312.02981/) (2023)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [gsplat: An Open-Source Library for Gaussian Splatting](https://mlsanity.com/paper/2409.06765/) (2024)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision](https://mlsanity.com/paper/2311.02077/) (2023)
- [3D Gaussian Splatting as Markov Chain Monte Carlo](https://mlsanity.com/paper/2404.09591/) (2024)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving](https://mlsanity.com/paper/2112.12610/) (2021)
- [OmniRe: Omni Urban Scene Reconstruction](https://mlsanity.com/paper/2408.16760/) (2024)
- [HUGS: Human Gaussian Splats](https://mlsanity.com/paper/2311.17910/) (2023)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)
- [SUDS: Scalable Urban Dynamic Scenes](https://mlsanity.com/paper/2303.14536/) (2023)
- [NeuRAD: Neural Rendering for Autonomous Driving](https://mlsanity.com/paper/2311.15260/) (2023)

## Built on by (most cited first, 30 of 69)

- [MTGS: Multi-Traversal Gaussian Splatting](https://mlsanity.com/paper/2503.12552/) (2025)
- [R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation](https://mlsanity.com/paper/2506.07826/) (2025)
- [PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Driving](https://mlsanity.com/paper/2507.17596/) (2025)
- [Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splatting](https://mlsanity.com/paper/2506.05280/) (2025)
- [GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention](https://mlsanity.com/paper/2505.10685/) (2025)
- [GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splatting](https://mlsanity.com/paper/2507.19451/) (2025)
- [PINGS: Gaussian Splatting Meets Distance Fields within a Point-Based Implicit Neural Map](https://mlsanity.com/paper/2502.05752/) (2025)
- [RealEngine: Simulating Autonomous Driving in Realistic Context](https://mlsanity.com/paper/2505.16902/) (2025)
- [Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2503.14537/) (2025)
- [RaGS: Unleashing 3D Gaussian Splatting from 4D Radar and Monocular Cues for 3D Object Detection](https://mlsanity.com/paper/2507.19856/) (2025)
- [Flux4D: Flow-based Unsupervised 4D Reconstruction](https://mlsanity.com/paper/2512.03210/) (2025)
- [LiHi-GS: LiDAR-Supervised Gaussian Splatting for Highway Driving Scene Reconstruction](https://mlsanity.com/paper/2412.15447/) (2024)
- [ParkGaussian: Surround-view 3D Gaussian Splatting for Autonomous Parking](https://mlsanity.com/paper/2601.01386/) (2026)
- [B'ezierGS: Dynamic Urban Scene Reconstruction with B'ezier Curve Gaussian Splatting](https://mlsanity.com/paper/2506.22099/) (2025)
- [Sim2Real Diffusion: Leveraging Foundation Vision Language Models for Adaptive Automated Driving](https://mlsanity.com/paper/2507.00236/) (2025)
- [LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Grounding](https://mlsanity.com/paper/2508.19204/) (2025)
- [Decomposing Densification in Gaussian Splatting for Faster 3D Scene Reconstruction](https://mlsanity.com/paper/2507.20239/) (2025)
- [ExtraGS: Geometric-Aware Trajectory Extrapolation with Uncertainty-Guided Generative Priors](https://mlsanity.com/paper/2508.15529/) (2025)
- [SimULi: Real-Time LiDAR and Camera Simulation with Unscented Transforms](https://mlsanity.com/paper/2510.12901/) (2025)
- [SaLF: Sparse Local Fields for Multi-Sensor Rendering in Real-Time](https://mlsanity.com/paper/2507.18713/) (2025)
- [RadarGaussianDet3D: Gaussian Representation-based Real-time 3D Object Detection with 4D Automotive Radars](https://mlsanity.com/paper/2509.16119/) (2025)
- [IDSplat: Instance-Decomposed 3D Gaussian Splatting for Driving Scenes](https://mlsanity.com/paper/2511.19235/) (2025)
- [Mirage: One-Step Video Diffusion for Photorealistic and Coherent Asset Editing in Driving Scenes](https://mlsanity.com/paper/2512.24227/) (2025)
- [UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modeling](https://mlsanity.com/paper/2602.20943/) (2026)
- [TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations](https://mlsanity.com/paper/2606.17386/) (2026)
- [2Xplat: Decoupling Geometry and Appearance Modeling for Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2603.21064/) (2026)
- [EnerGS: Energy-Based Gaussian Splatting with Partial Geometric Priors](https://mlsanity.com/paper/2604.26238/) (2026)
- [Hybrid Rendering for Multimodal Autonomous Driving: Merging Neural and Physics-Based Simulation](https://mlsanity.com/paper/2503.09464/) (2025)
- [LidarPainter: One-Step Away From Any Lidar View To Novel Guidance](https://mlsanity.com/paper/2507.12114/) (2025)
- [Physically Based Neural LiDAR Resimulation](https://mlsanity.com/paper/2507.12489/) (2025)

## Abstract (arXiv)

Ensuring the safety of autonomous robots, such as self-driving vehicles, requires extensive testing across diverse driving scenarios. Simulation is a key ingredient for conducting such testing in a cost-effective and scalable way. Neural rendering methods have gained popularity, as they can build simulation environments from collected logs in a data-driven manner. However, existing neural radiance field (NeRF) methods for sensor-realistic rendering of camera and lidar data suffer from low rendering speeds, limiting their applicability for large-scale testing. While 3D Gaussian Splatting (3DGS) enables real-time rendering, current methods are limited to camera data and are unable to render lidar data essential for autonomous driving. To address these limitations, we propose SplatAD, the first 3DGS-based method for realistic, real-time rendering of dynamic scenes for both camera and lidar data. SplatAD accurately models key sensor-specific phenomena such as rolling shutter effects, lidar intensity, and lidar ray dropouts, using purpose-built algorithms to optimize rendering efficiency. Evaluation across three autonomous driving datasets demonstrates that SplatAD achieves state-of-the-art rendering quality with up to +2 PSNR for NVS and +3 PSNR for reconstruction while increasing rendering speed over NeRF-based methods by an order of magnitude. See https://research.zenseact.com/publications/splatad/ for our project page.