mlsanityData to 7 Oct 2026

Vision-based 3D occupancy prediction in autonomous driving: a review and outlook

arXiv 2405.02595 ↗First posted 4 May 2024cs.CVcited by 29

Not on the map. Why?

Abstract (arXiv)

In recent years, autonomous driving has garnered escalating attention for its potential to relieve drivers' burdens and improve driving safety. Vision-based 3D occupancy prediction, which predicts the spatial occupancy status and semantics of 3D voxel grids around the autonomous vehicle from image inputs, is an emerging perception task suitable for cost-effective perception system of autonomous driving. Although numerous studies have demonstrated the greater advantages of 3D occupancy prediction over object-centric perception tasks, there is still a lack of a dedicated review focusing on this rapidly developing field. In this paper, we first introduce the background of vision-based 3D occupancy prediction and discuss the challenges in this task. Secondly, we conduct a comprehensive survey of the progress in vision-based 3D occupancy prediction from three aspects: feature enhancement, deployment friendliness and label efficiency, and provide an in-depth analysis of the potentials and challenges of each category of methods. Finally, we present a summary of prevailing research trends and propose some inspiring future outlooks. To provide a valuable reference for researchers, a regularly updated collection of related papers, datasets, and codes is organized at https://github.com/zya3d/Awesome-3D-Occupancy-Prediction.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 75 of its 100 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

75 of 100 references.

References · 75

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  9. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  10. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  11. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  12. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
38 more
  1. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  2. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  3. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  4. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  5. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  6. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  7. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  8. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 409
  9. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  10. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 357
  11. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  12. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  13. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 306
  14. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 287
  15. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  16. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 222
  17. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 218
  18. 2023Scene as Occupancycited by 208
  19. 2020LMSCNet: Lightweight Multiscale 3D Semantic Completioncited by 187
  20. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  21. 2020EPNet: Enhancing Point Features with Image Semantics for 3D Object Detectioncited by 164
  22. 2019Stereo R-CNN based 3D Object Detection for Autonomous Drivingcited by 163
  23. 2023FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformationcited by 162
  24. 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
  25. 2023SelfOcc: Self-Supervised Vision-Based 3D Occupancy Predictioncited by 133
  26. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 126
  27. 2022Vision-Centric BEV Perception: A Surveycited by 126
  28. 2023RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervisioncited by 126
  29. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 124
  30. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 114
  31. 2023FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugincited by 101
  32. 2020DSGN: Deep Stereo Geometry Network for 3D Object Detectioncited by 100
  33. 2023FB-BEV: BEV Representation from Forward-Backward View Transformationscited by 98
  34. 2017Deep MANTA: A Coarse-to-fine Many-Task Network for joint 2D and 3D vehicle analysis from monocular imagecited by 95
  35. 2023COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Predictioncited by 94
  36. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  37. 2023Symphonize 3D Semantic Scene Completion with Contextual Instance Queriescited by 88
  38. 2023OccDepth: A Depth-Aware Method for 3D Semantic Scene Completioncited by 85

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 17; 2026: 6.

29 papers.

Sort:

By area:

  1. 2025A Survey of World Models for Autonomous Drivingcited by 60
  2. 2024A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspectivecited by 37
  3. 2024OccMamba: Semantic Occupancy Prediction with State Space Modelscited by 29
  4. 2025UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Drivingcited by 27
  5. 2025GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flowcited by 24
  6. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  7. 2024LangOcc: Self-Supervised Open Vocabulary Occupancy Estimation via Volume Renderingcited by 16
  8. 2025GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attentioncited by 14
  9. 2025Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Predictioncited by 9
  10. 2024ViPOcc: Leveraging Visual Priors from Vision Foundation Models for Single-View 3D Occupancy Predictioncited by 8
  11. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  12. 2024LightOcc: Lightweight Spatial Embedding for Efficient Vision-based 3D Occupancy Predictioncited by 6
17 more
  1. 2025OCCUQ: Exploring Efficient Uncertainty Quantification for 3D Occupancy Predictioncited by 6
  2. 2025OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspectivecited by 5
  3. 2025Progressive Gaussian Transformer with Anisotropy-aware Sampling for Open Vocabulary Occupancy Predictioncited by 4
  4. 2025Event-aided Semantic Scene Completioncited by 3
  5. 2025IPFormer: Visual 3D Panoptic Scene Completion with Context-Adaptive Instance Proposalscited by 3
  6. 2025SA-Occ: Satellite-Assisted 3D Occupancy Prediction in Real Worldcited by 2
  7. 2025Camera-Only 3D Panoptic Scene Completion for Autonomous Driving through Differentiable Object Shapescited by 2
  8. 2025S2GO: Streaming Sparse Gaussian Occupancy Predictioncited by 1
  9. 2025A Synthetic Benchmark for Collaborative 3D Semantic Occupancy Prediction in V2X-Enabled Autonomous Drivingcited by 1
  10. 2026VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Predictioncited by 1
  11. 2025DGOcc: Depth-aware Global Query-based Network for Monocular 3D Occupancy Predictioncited by 0
  12. 2025TACOcc:Target-Adaptive Cross-Modal Fusion with Volume Rendering for 3D Semantic Occupancycited by 0
  13. 2026Rebenchmarking Unsupervised Monocular 3D Occupancy Predictioncited by 0
  14. 2026BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusioncited by 0
  15. 2026TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarioscited by 0
  16. 2026Group-wise Supervision with Focal-Dice Loss for Long-Tailed Indoor Semantic Occupancy Predictioncited by 0
  17. 2026LetOccVote: Learning Weakly Supervised 3D Occupancy through Consensuscited by 0