mlsanityData to 7 Oct 2026

SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction

arXiv 2311.12754 ↗First posted 21 Nov 2023cs.CV cs.AI cs.LGcited by 133

Not on the map. Why?

Abstract (arXiv)

3D occupancy prediction is an important task for the robustness of vision-centric autonomous driving, which aims to predict whether each point is occupied in the surrounding 3D space. Existing methods usually require 3D occupancy labels to produce meaningful results. However, it is very laborious to annotate the occupancy status of each voxel. In this paper, we propose SelfOcc to explore a self-supervised way to learn 3D occupancy using only video sequences. We first transform the images into the 3D space (e.g., bird's eye view) to obtain 3D representation of the scene. We directly impose constraints on the 3D representations by treating them as signed distance fields. We can then render 2D images of previous and future frames as self-supervision signals to learn the 3D representations. We propose an MVS-embedded strategy to directly optimize the SDF-induced weights with multiple depth proposals. Our SelfOcc outperforms the previous best method SceneRF by 58.7% using a single frame as input on SemanticKITTI and is the first self-supervised work that produces reasonable 3D occupancy for surround cameras on nuScenes. SelfOcc produces high-quality depth and achieves state-of-the-art results on novel depth synthesis, monocular depth estimation, and surround-view depth estimation on the SemanticKITTI, KITTI-2015, and nuScenes, respectively. Code: https://github.com/huang-yh/SelfOcc.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 76 of its 89 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

76 of 89 references.

References · 76

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  6. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  7. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  8. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  9. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  10. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  11. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  12. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
38 more
  1. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  2. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  3. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  4. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 732
  5. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  6. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  7. 2020Implicit Geometric Regularization for Learning Shapescited by 707
  8. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  9. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  10. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  11. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  12. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  13. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  14. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  15. 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 437
  16. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  17. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  18. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 357
  19. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  20. 2021Dense Depth Priors for Neural Radiance Fields from Sparse Input Viewscited by 301
  21. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 287
  22. 2021Urban Radiance Fieldscited by 263
  23. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  24. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 222
  25. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 218
  26. 2023Scene as Occupancycited by 208
  27. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  28. 2021NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereocited by 191
  29. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  30. 2020LMSCNet: Lightweight Multiscale 3D Semantic Completioncited by 187
  31. 2023FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformationcited by 162
  32. 2018Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstructioncited by 159
  33. 2021Offboard 3D Object Detection from Point Cloud Sequencescited by 144
  34. 2020Feature-metric Loss for Self-supervised Learning of Depth and Egomotioncited by 140
  35. 2021MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesiscited by 133
  36. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 126
  37. 2023RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervisioncited by 126
  38. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 124

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 45; 2025: 55; 2026: 30.

133 papers.

Sort:

By area:

  1. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  2. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  3. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 155
  4. 2024GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Predictioncited by 83
  5. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  6. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  7. 2025A Survey of World Models for Autonomous Drivingcited by 60
  8. 2024DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Inputcited by 53
  9. 2024GaussianOcc: Fully Self-supervised and Efficient 3D Occupancy Estimation with Gaussian Splattingcited by 50
  10. 2023OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queriescited by 48
  11. 2024GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understandingcited by 46
  12. 2024RenderWorld: World Model with Self-Supervised 3D Labelcited by 44
38 more
  1. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 43
  2. 2023OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environmentscited by 39
  3. 2025RAP: 3D Rasterization Augmented End-to-End Planningcited by 39
  4. 2024A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspectivecited by 37
  5. 2024GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generationcited by 35
  6. 2024DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Featurescited by 33
  7. 2024RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radarcited by 32
  8. 2024Vision-based 3D occupancy prediction in autonomous driving: a review and outlookcited by 29
  9. 2024Monocular Occupancy Prediction for Scalable Indoor Scenescited by 29
  10. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  11. 2025GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flowcited by 24
  12. 2024Let Occ Flow: Self-Supervised 3D Occupancy Flow Predictioncited by 23
  13. 2024OccFlowNet: Towards Self-supervised Occupancy Estimation via Differentiable Rendering and Occupancy Flowcited by 22
  14. 2024VEON: Vocabulary-Enhanced Occupancy Predictioncited by 22
  15. 2024PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priorscited by 21
  16. 2024GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervisioncited by 21
  17. 2025DVGT: Driving Visual Geometry Transformercited by 20
  18. 2024ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictionscited by 19
  19. 2025QuadricFormer: Scene as Superquadrics for 3D Semantic Occupancy Predictioncited by 19
  20. 2024Language Driven Occupancy Predictioncited by 17
  21. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  22. 2025GaussRender: Learning 3D Occupancy with Gaussian Renderingcited by 17
  23. 2024LangOcc: Self-Supervised Open Vocabulary Occupancy Estimation via Volume Renderingcited by 16
  24. 2024Deep Height Decoupling for Precise Vision-based 3D Occupancy Predictioncited by 16
  25. 2025AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splattingcited by 14
  26. 2025GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attentioncited by 14
  27. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 14
  28. 2024Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Drivingcited by 12
  29. 2024HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Predictioncited by 12
  30. 2024GaussianPretrain: A Simple Unified 3D Gaussian Representation for Visual Pre-training in Autonomous Drivingcited by 11
  31. 2025Feed-Forward SceneDINO for Unsupervised Semantic Scene Completioncited by 11
  32. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11
  33. 2024OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generationcited by 10
  34. 2025SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queriescited by 10
  35. 20253D Occupancy Prediction with Low-Resolution Queries via Prototype-aware View Transformationcited by 9
  36. 2025AGO: Adaptive Grounding for Open World 3D Occupancy Predictioncited by 9
  37. 2025VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Predictioncited by 9
  38. 2025OccTENS: 3D Occupancy World Model via Temporal Next-Scale Predictioncited by 9

50 most cited of 133 All 133 (JSON)