mlsanityData to 7 Oct 2026

VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion

arXiv 2302.12251 ↗First posted 23 Feb 2023cs.CV cs.AI cs.LGcited by 306

Not on the map. Why?

Cited by 21 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and understanding. To enable such capability in AI systems, we propose VoxFormer, a Transformer-based semantic scene completion framework that can output complete 3D volumetric semantics from only 2D images. Our framework adopts a two-stage design where we start from a sparse set of visible and occupied voxel queries from depth estimation, followed by a densification stage that generates dense 3D voxels from the sparse ones. A key idea of this design is that the visual features on 2D images correspond only to the visible scene structures rather than the occluded or empty spaces. Therefore, starting with the featurization and prediction of the visible structures is more reliable. Once we obtain the set of sparse queries, we apply a masked autoencoder design to propagate the information to all the voxels by self-attention. Experiments on SemanticKITTI show that VoxFormer outperforms the state of the art with a relative improvement of 20.0% in geometry and 18.1% in semantics and reduces GPU memory during training to less than 16GB. Our code is available on https://github.com/NVlabs/VoxFormer.

Key facts. Cited by 306 papers in the map (data through 7 Oct 2026). Builds on 64 of its 77 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

64 of 77 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  12. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
38 more
  1. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  2. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  3. 2020Convolutional Occupancy Networkscited by 760
  4. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  5. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  6. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  7. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  8. 2021NICE-SLAM: Neural Implicit Scalable Encoding for SLAMcited by 586
  9. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 584
  10. 2019Deep Learning for 3D Point Clouds: A Surveycited by 557
  11. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  12. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  13. 2019PANet: Few-Shot Image Semantic Segmentation with Prototype Alignmentcited by 449
  14. 2018PCN: Point Completion Networkcited by 382
  15. 2015Learning Deconvolution Network for Semantic Segmentationcited by 365
  16. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 357
  17. 2020Implicit Functions in Feature Space for 3D Shape Reconstruction and Completioncited by 343
  18. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
  19. 2022V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformercited by 320
  20. 2020V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Predictioncited by 292
  21. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
  22. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 222
  23. 2016Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesiscited by 219
  24. 2021Learning Distilled Collaboration Graph for Multi-Agent Perceptioncited by 194
  25. 2020LMSCNet: Lightweight Multiscale 3D Semantic Completioncited by 187
  26. 2017Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistencycited by 177
  27. 2022V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Drivingcited by 157
  28. 2022M2BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representationcited by 155
  29. 2022Rethinking Semantic Segmentation: A Prototype Viewcited by 140
  30. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 126
  31. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 124
  32. 2020HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matchingcited by 120
  33. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  34. 2016Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervisioncited by 106
  35. 2022ShapeFormer: Transformer-based Shape Completion via Sparse Representationcited by 106
  36. 2020S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Cloudscited by 102
  37. 2020A Survey on Deep Learning Techniques for Stereo-based Depth Estimationcited by 90
  38. 2019Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Eracited by 87

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 41; 2024: 95; 2025: 102; 2026: 68.

306 papers.

Sort:

By area:

  1. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  2. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  3. 2023Scene as Occupancycited by 208
  4. 2023V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perceptioncited by 155
  5. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 155
  6. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  7. 2023RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervisioncited by 126
  8. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 114
  9. 2023FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugincited by 101
  10. 2023COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Predictioncited by 94
  11. 2024SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Predictioncited by 94
  12. 2023Symphonize 3D Semantic Scene Completion with Contextual Instance Queriescited by 88
38 more
  1. 2023NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Spacecited by 88
  2. 2024GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Predictioncited by 83
  3. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  4. 2023SSCBench: A Large-Scale 3D Semantic Scene Completion Benchmark for Autonomous Drivingcited by 72
  5. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  6. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  7. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  8. 2023Fully Sparse 3D Occupancy Predictioncited by 61
  9. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Modelcited by 58
  10. 2024OPUS: Occupancy Prediction Using a Sparse Setcited by 56
  11. 2024SCube: Instant Large-Scale Scene Reconstruction using VoxSplatscited by 56
  12. 2024SemCity: Semantic Scene Generation with Triplane Diffusioncited by 53
  13. 2024FastOcc: Accelerating 3D Occupancy Prediction by Fusing the 2D Bird's-Eye View and Perspective Viewcited by 51
  14. 2023Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applicationscited by 50
  15. 2024Context and Geometry Aware Voxel Transformer for Semantic Scene Completioncited by 50
  16. 2023OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queriescited by 48
  17. 2024GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understandingcited by 46
  18. 2024MonoOcc: Digging into Monocular Semantic Occupancy Predictioncited by 43
  19. 2024Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillationcited by 43
  20. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 43
  21. 2023PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awarenesscited by 42
  22. 2023Camera-based 3D Semantic Scene Completion with Sparse Guidance Networkcited by 40
  23. 2023OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environmentscited by 39
  24. 2023Among Us: Adversarially Robust Collaborative Perception by Consensuscited by 37
  25. 2024A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspectivecited by 37
  26. 2024ViewFormer: Exploring Spatiotemporal Modeling for Multi-View 3D Occupancy Perception via View-Guided Transformerscited by 36
  27. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  28. 2024Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehiclescited by 33
  29. 2023UniWorld: Autonomous Driving Pre-training via World Modelscited by 32
  30. 2024OccFusion: Multi-Sensor Fusion Framework for 3D Semantic Occupancy Predictioncited by 32
  31. 2024RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radarcited by 32
  32. 2024Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completioncited by 32
  33. 2023DepthSSC: Monocular 3D Semantic Scene Completion via Depth-Spatial Alignment and Voxel Adaptationcited by 30
  34. 2024Vision-based 3D occupancy prediction in autonomous driving: a review and outlookcited by 29
  35. 2024Monocular Occupancy Prediction for Scalable Indoor Scenescited by 29
  36. 2024OccMamba: Semantic Occupancy Prediction with State Space Modelscited by 29
  37. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  38. 2024OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Drivingcited by 28

50 most cited of 306 All 306 (JSON)