# SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction

arXiv: https://arxiv.org/abs/2404.09502 · HTML: https://mlsanity.com/paper/2404.09502/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2304.05316/) (2023)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion](https://mlsanity.com/paper/2012.03762/) (2020)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [Scene as Occupancy](https://mlsanity.com/paper/2306.02851/) (2023)
- [LMSCNet: Lightweight Multiscale 3D Semantic Completion](https://mlsanity.com/paper/2008.10559/) (2020)
- [Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks](https://mlsanity.com/paper/2003.13402/) (2020)
- [2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds](https://mlsanity.com/paper/2207.04397/) (2022)

## Built on by (most cited first, 30 of 97)

- [GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction](https://mlsanity.com/paper/2412.04384/) (2024)
- [OPUS: Occupancy Prediction Using a Sparse Set](https://mlsanity.com/paper/2409.09350/) (2024)
- [A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective](https://mlsanity.com/paper/2405.05173/) (2024)
- [OmniNWM: Omniscient Driving Navigation World Models](https://mlsanity.com/paper/2510.18313/) (2025)
- [OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2404.15014/) (2024)
- [OccMamba: Semantic Occupancy Prediction with State Space Models](https://mlsanity.com/paper/2408.09859/) (2024)
- [OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving](https://mlsanity.com/paper/2412.10734/) (2024)
- [GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flow](https://mlsanity.com/paper/2502.17288/) (2025)
- [Let Occ Flow: Self-Supervised 3D Occupancy Flow Prediction](https://mlsanity.com/paper/2407.07587/) (2024)
- [STCOcc: Sparse Spatial-Temporal Cascade Renovation for 3D Occupancy and Scene Flow Prediction](https://mlsanity.com/paper/2504.19749/) (2025)
- [DVGT: Driving Visual Geometry Transformer](https://mlsanity.com/paper/2512.16919/) (2025)
- [GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision](https://mlsanity.com/paper/2405.10591/) (2024)
- [QuadricFormer: Scene as Superquadrics for 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2506.10977/) (2025)
- [ProtoOcc: Accurate, Efficient 3D Occupancy Prediction Using Dual Branch Encoder-Prototype Query Decoder](https://mlsanity.com/paper/2412.08774/) (2024)
- [DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Driving](https://mlsanity.com/paper/2507.04049/) (2025)
- [SafeMap: Robust HD Map Construction from Incomplete Observations](https://mlsanity.com/paper/2507.00861/) (2025)
- [VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction](https://mlsanity.com/paper/2506.05563/) (2025)
- [Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Prediction](https://mlsanity.com/paper/2602.21552/) (2026)
- [OccRWKV: Rethinking Efficient 3D Semantic Occupancy Prediction with Linear Complexity](https://mlsanity.com/paper/2409.19987/) (2024)
- [Skip Mamba Diffusion for Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2501.07260/) (2025)
- [Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2503.14537/) (2025)
- [SplatSSC: Decoupled Depth-Guided Gaussian Splatting for Semantic Scene Completion](https://mlsanity.com/paper/2508.02261/) (2025)
- [EFFOcc: Learning Efficient Occupancy Networks from Minimal Labels for Autonomous Driving](https://mlsanity.com/paper/2406.07042/) (2024)
- [RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments](https://mlsanity.com/paper/2408.15503/) (2024)
- [SliceOcc: Indoor 3D Semantic Occupancy Prediction with Vertical Slice Representation](https://mlsanity.com/paper/2501.16684/) (2025)
- [Self-Supervised Sparse Sensor Fusion for Long Range Perception](https://mlsanity.com/paper/2508.13995/) (2025)
- [OneOcc: Semantic Occupancy Prediction for Legged Robots with a Single Panoramic Camera](https://mlsanity.com/paper/2511.03571/) (2025)
- [ReliOcc: Towards Reliable Semantic Occupancy Prediction via Uncertainty Learning](https://mlsanity.com/paper/2409.18026/) (2024)
- [Robust 3D Semantic Occupancy Prediction with Calibration-free Spatial Transformation](https://mlsanity.com/paper/2411.12177/) (2024)
- [GSRender: Deduplicated Occupancy Prediction via Weakly Supervised 3D Gaussian Splatting](https://mlsanity.com/paper/2412.14579/) (2024)

## Abstract (arXiv)

Vision-based perception for autonomous driving requires an explicit modeling of a 3D space, where 2D latent representations are mapped and subsequent 3D operators are applied. However, operating on dense latent spaces introduces a cubic time and space complexity, which limits scalability in terms of perception range or spatial resolution. Existing approaches compress the dense representation using projections like Bird's Eye View (BEV) or Tri-Perspective View (TPV). Although efficient, these projections result in information loss, especially for tasks like semantic occupancy prediction. To address this, we propose SparseOcc, an efficient occupancy network inspired by sparse point cloud processing. It utilizes a lossless sparse latent representation with three key innovations. Firstly, a 3D sparse diffuser performs latent completion using spatially decomposed 3D sparse convolutional kernels. Secondly, a feature pyramid and sparse interpolation enhance scales with information from others. Finally, the transformer head is redesigned as a sparse variant. SparseOcc achieves a remarkable 74.9% reduction on FLOPs over the dense baseline. Interestingly, it also improves accuracy, from 12.8% to 14.1% mIOU, which in part can be attributed to the sparse representation's ability to avoid hallucinations on empty voxels.