mlsanityData to 7 Oct 2026

RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision

arXiv 2309.09502 ↗First posted 18 Sep 2023cs.CVcited by 126

Not on the map. Why?

Abstract (arXiv)

3D occupancy prediction holds significant promise in the fields of robot perception and autonomous driving, which quantifies 3D scenes into grid cells with semantic labels. Recent works mainly utilize complete occupancy labels in 3D voxel space for supervision. However, the expensive annotation process and sometimes ambiguous labels have severely constrained the usability and scalability of 3D occupancy models. To address this, we present RenderOcc, a novel paradigm for training 3D occupancy models only using 2D labels. Specifically, we extract a NeRF-style 3D volume representation from multi-view images, and employ volume rendering techniques to establish 2D renderings, thus enabling direct 3D supervision from 2D semantics and depth labels. Additionally, we introduce an Auxiliary Ray method to tackle the issue of sparse viewpoints in autonomous driving scenarios, which leverages sequential frames to construct comprehensive 2D rendering for each object. To our best knowledge, RenderOcc is the first attempt to train multi-view 3D occupancy models only using 2D labels, reducing the dependence on costly 3D occupancy annotations. Extensive experiments demonstrate that RenderOcc achieves comparable performance to models fully supervised with 3D labels, underscoring the significance of this approach in real-world applications.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

39 of 51 references.

References · 39

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  5. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  6. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  7. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  8. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  9. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  10. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  11. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  12. 2020Center-based 3D Object Detection and Trackingcited by 1,143
27 more
  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  3. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  4. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  5. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  6. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  7. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  8. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  9. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 444
  10. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  11. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 357
  12. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  13. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  14. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  15. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 306
  16. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 287
  17. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 218
  18. 2023Scene as Occupancycited by 208
  19. 2020LMSCNet: Lightweight Multiscale 3D Semantic Completioncited by 187
  20. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 177
  21. 2022Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusioncited by 161
  22. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 126
  23. 2022BEVStereo: Enhancing Depth Estimation in Multi-view 3D Object Detection with Dynamic Temporal Stereocited by 89
  24. 2023OccDepth: A Depth-Aware Method for 3D Semantic Scene Completioncited by 85
  25. 2022SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimationcited by 76
  26. 2023Behind the Scenes: Density Fields for Single View Reconstructioncited by 63
  27. 2022BEV-SAN: Accurate BEV 3D Object Detection via Slice Attention Networkscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 40; 2025: 53; 2026: 26.

126 papers.

Sort:

By area:

  1. 2023SelfOcc: Self-Supervised Vision-Based 3D Occupancy Predictioncited by 133
  2. 2023FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugincited by 101
  3. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  4. 2023Fully Sparse 3D Occupancy Predictioncited by 61
  5. 2025A Survey of World Models for Autonomous Drivingcited by 60
  6. 2024OPUS: Occupancy Prediction Using a Sparse Setcited by 56
  7. 2024GaussianOcc: Fully Self-supervised and Efficient 3D Occupancy Estimation with Gaussian Splattingcited by 50
  8. 2023OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queriescited by 48
  9. 2024Co-Occ: Coupling Explicit Feature Fusion with Volume Rendering Regularization for Multi-Modal 3D Semantic Occupancy Predictioncited by 46
  10. 2024GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understandingcited by 46
  11. 2024RenderWorld: World Model with Self-Supervised 3D Labelcited by 44
  12. 2023OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environmentscited by 39
38 more
  1. 2024A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspectivecited by 37
  2. 2024Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perceptioncited by 35
  3. 2024GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generationcited by 35
  4. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  5. 2024OccFusion: Multi-Sensor Fusion Framework for 3D Semantic Occupancy Predictioncited by 32
  6. 2024Vision-based 3D occupancy prediction in autonomous driving: a review and outlookcited by 29
  7. 2024Monocular Occupancy Prediction for Scalable Indoor Scenescited by 29
  8. 2024OccMamba: Semantic Occupancy Prediction with State Space Modelscited by 29
  9. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  10. 2025GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flowcited by 24
  11. 2024Any2Point: Empowering Any-modality Large Models for Efficient 3D Understandingcited by 23
  12. 2024Let Occ Flow: Self-Supervised 3D Occupancy Flow Predictioncited by 23
  13. 2024OccFlowNet: Towards Self-supervised Occupancy Estimation via Differentiable Rendering and Occupancy Flowcited by 22
  14. 2024GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervisioncited by 21
  15. 2025STCOcc: Sparse Spatial-Temporal Cascade Renovation for 3D Occupancy and Scene Flow Predictioncited by 20
  16. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  17. 2024Neural Radiance Field in Autonomous Driving: A Surveycited by 19
  18. 2024ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictionscited by 19
  19. 2024Language Driven Occupancy Predictioncited by 17
  20. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 17
  21. 2025GaussRender: Learning 3D Occupancy with Gaussian Renderingcited by 17
  22. 2024LangOcc: Self-Supervised Open Vocabulary Occupancy Estimation via Volume Renderingcited by 16
  23. 2024Deep Height Decoupling for Precise Vision-based 3D Occupancy Predictioncited by 16
  24. 2024DAOcc: 3D Object Detection Assisted Multi-Sensor Fusion for 3D Occupancy Predictioncited by 15
  25. 2024VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Drivingcited by 15
  26. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  27. 2024Panoptic-FlashOcc: An Efficient Baseline to Marry Semantic Occupancy with Panoptic via Instance Centercited by 14
  28. 2025AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splattingcited by 14
  29. 2025GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attentioncited by 14
  30. 2024OccFusion: Depth Estimation Free Multi-sensor Fusion for 3D Occupancy Predictioncited by 13
  31. 2024Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Drivingcited by 12
  32. 2024HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Predictioncited by 12
  33. 2025Feed-Forward SceneDINO for Unsupervised Semantic Scene Completioncited by 11
  34. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11
  35. 2025SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queriescited by 10
  36. 2025Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Predictioncited by 9
  37. 2025VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Predictioncited by 9
  38. 2025OccTENS: 3D Occupancy World Model via Temporal Next-Scale Predictioncited by 9

50 most cited of 126 All 126 (JSON)