mlsanityData to 7 Oct 2026

Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey

arXiv 2503.14537 ↗First posted 17 Mar 2025cs.CV cs.ROcited by 9

On the map: End-to-end autonomous driving → Generative simulation for driving

Abstract (arXiv)

Learning-based 3D reconstruction has emerged as a transformative technique in autonomous driving, enabling precise modeling of environments through advanced neural representations. It has inspired pioneering solutions for vital tasks in autonomous driving, such as dense mapping and closed-loop simulation, as well as comprehensive scene feature for driving scene understanding and reasoning. Given the rapid growth in related research, this survey provides a comprehensive review of both technical evolutions and practical applications in autonomous driving. We begin with an introduction to the preliminaries of learning-based 3D reconstruction to provide a solid technical background foundation, then progress to a rigorous, multi-dimensional examination of cutting-edge methodologies, systematically organized according to the distinctive technical requirements and fundamental challenges of autonomous driving. Through analyzing and summarizing development trends and cutting-edge research, we identify existing technical challenges, along with insufficient disclosure of on-board validation and safety verification details in the current literature, and ultimately suggest potential directions to guide future studies.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving. Builds on 204 of its 285 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

204 of 285 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation24 Jan
  2. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers24 Dec
  3. 2024GaussianAD: Gaussian-Centric End-to-End Autonomous Driving13 Dec
  4. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving2 Dec
  5. 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving2 Dec
  6. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation17 Oct
  7. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving1 Aug
  8. 2024XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis26 Jun
  9. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation3 Jun
  10. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  11. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
  12. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
4 more
  1. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents8 Feb
  2. 2024OASim: an Open and Adaptive Simulator based on Neural Rendering for Autonomous Driving6 Feb
  3. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  4. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep

Other references · 188

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2023Segment Anythingcited by 8,158
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
38 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  2. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  7. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  8. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  9. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  10. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  11. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  12. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  13. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  14. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  15. 2024Depth Anything V2cited by 1,385
  16. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  17. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  18. 2022Planning-oriented Autonomous Drivingcited by 1,068
  19. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  20. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  21. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  22. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  23. 2024Genie: Generative Interactive Environmentscited by 706
  24. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  25. 2023Scaffold-GS: Structured 3D Gaussians for View-Adaptive Renderingcited by 684
  26. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  27. 2023Mip-Splatting: Alias-free 3D Gaussian Splattingcited by 661
  28. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  29. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  30. 2022Point-NeRF: Point-based Neural Radiance Fieldscited by 503
  31. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  32. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  33. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 396
  34. 2019Panoptic-DeepLabcited by 362
  35. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  36. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 359
  37. 2020Virtual KITTI 2cited by 345
  38. 2020Neural Scene Graphs for Dynamic Scenescited by 298

50 most cited of 188 All 188 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2026MeMix: Writing Less, Remembering More for Streaming 3D Reconstructioncited by 5
  2. 2025LiDAR-GS++:Improving LiDAR Gaussian Reconstruction via Diffusion Priorscited by 2
  3. 2025SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restorationcited by 2
  4. 2025DIAL-GS: Dynamic Instance Aware Reconstruction for Label-free Street Scenes with 4D Gaussian Splattingcited by 1
  5. 2026Stitch4D: Sparse Multi-Location 4D Urban Reconstruction via Spatio-Temporal Interpolationcited by 0
  6. 2026Empowering Feed-Forward Reconstruction Models with Metric Scale via Satellite Imagescited by 0
  7. 2026RoadWeaver: Large-Scale Lane-Level HD Map Generation from Scratch for Autonomous Driving Simulationcited by 0
  8. 2026RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actorscited by 0
  9. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Review of 3D Object Detection in Autonomous Driving: Technological Advances and Future Directions
  2. 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
  3. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale
  4. 2023End-to-end Autonomous Driving: Challenges and Frontiers
  5. 2025VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
  6. 2025LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Grounding
  7. 2026UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modeling
  8. 2025A Survey of World Models for Autonomous Driving