mlsanityData to 14 Oct 2026

GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction

arXiv 2405.17429 ↗First posted 27 May 2024cs.CV cs.AIcited by 157

Not on the map. Why?

Abstract (arXiv)

3D semantic occupancy prediction aims to obtain 3D fine-grained geometry and semantics of the surrounding scene and is an important task for the robustness of vision-centric autonomous driving. Most existing methods employ dense grids such as voxels as scene representations, which ignore the sparsity of occupancy and the diversity of object scales and thus lead to unbalanced allocation of resources. To address this, we propose an object-centric representation to describe 3D scenes with sparse 3D semantic Gaussians where each Gaussian represents a flexible region of interest and its semantic features. We aggregate information from images through the attention mechanism and iteratively refine the properties of 3D Gaussians including position, covariance, and semantics. We then propose an efficient Gaussian-to-voxel splatting method to generate 3D occupancy predictions, which only aggregates the neighboring Gaussians for a certain position. We conduct extensive experiments on the widely adopted nuScenes and KITTI-360 datasets. Experimental results demonstrate that GaussianFormer achieves comparable performance with state-of-the-art methods with only 17.8% - 24.8% of their memory consumption. Code is available at: https://github.com/huang-yh/GaussianFormer.

Key facts. Cited by 157 papers in the map (data through 14 Oct 2026). Builds on 55 of its 64 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

55 of 64 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  7. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  8. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  9. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  10. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  11. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  12. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
38 more
  1. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  2. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 759
  3. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  4. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  5. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 684
  6. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  7. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  8. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 572
  9. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
  10. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 447
  11. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
  12. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 363
  13. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 316
  14. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
  15. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  16. 2023Text-to-3D using Gaussian Splattingcited by 299
  17. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 293
  18. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  19. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 258
  20. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 233
  21. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  22. 2024A Survey on 3D Gaussian Splattingcited by 224
  23. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 223
  24. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 221
  25. 2023Scene as Occupancycited by 209
  26. 2020Atlas: End-to-End 3D Scene Reconstruction from Posed Imagescited by 206
  27. 2020LMSCNet: Lightweight Multiscale 3D Semantic Completioncited by 188
  28. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 178
  29. 2023FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformationcited by 165
  30. 2022PolarFormer: Multi-camera 3D Object Detection with Polar Transformercited by 147
  31. 2022NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editingcited by 142
  32. 2023SelfOcc: Self-Supervised Vision-Based 3D Occupancy Predictioncited by 133
  33. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 127
  34. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 126
  35. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 115
  36. 2023FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugincited by 103
  37. 2023Symphonize 3D Semantic Scene Completion with Contextual Instance Queriescited by 90
  38. 2023OccDepth: A Depth-Aware Method for 3D Semantic Scene Completioncited by 86

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 61; 2026: 75.

157 papers.

Sort:

By area:

  1. 2024GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Predictioncited by 86
  2. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 67
  3. 2025A Survey of World Models for Autonomous Drivingcited by 60
  4. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  5. 2024GaussianOcc: Fully Self-supervised and Efficient 3D Occupancy Estimation with Gaussian Splattingcited by 50
  6. 2024RenderWorld: World Model with Self-Supervised 3D Labelcited by 46
  7. 2024GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understandingcited by 46
  8. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 45
  9. 2024GaussianAD: Gaussian-Centric End-to-End Autonomous Drivingcited by 31
  10. 2025GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flowcited by 24
  11. 2025DVGT: Driving Visual Geometry Transformercited by 21
  12. 2024ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictionscited by 20
38 more
  1. 2025QuadricFormer: Scene as Superquadrics for 3D Semantic Occupancy Predictioncited by 20
  2. 2024PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Viewscited by 19
  3. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Trainingcited by 18
  4. 2025GaussRender: Learning 3D Occupancy with Gaussian Renderingcited by 18
  5. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  6. 2024Deep Height Decoupling for Precise Vision-based 3D Occupancy Predictioncited by 16
  7. 2025GaussianFusion: Gaussian-Based Multi-Sensor Fusion for End-to-End Autonomous Drivingcited by 16
  8. 2024VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Drivingcited by 15
  9. 2025GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attentioncited by 14
  10. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 13
  11. 2024GaussianPretrain: A Simple Unified 3D Gaussian Representation for Visual Pre-training in Autonomous Drivingcited by 11
  12. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11
  13. 2026Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Predictioncited by 11
  14. 2025VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Predictioncited by 10
  15. 2025UniSplat: Unified Spatio-Temporal Fusion via 3D Latent Scaffolds for Dynamic Driving Scene Reconstructioncited by 10
  16. 2025Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Predictioncited by 9
  17. 2025RaGS: Unleashing 3D Gaussian Splatting from 4D Radar and Monocular Cues for 3D Object Detectioncited by 9
  18. 2025SplatSSC: Decoupled Depth-Guided Gaussian Splatting for Semantic Scene Completioncited by 9
  19. 2025SliceOcc: Indoor 3D Semantic Occupancy Prediction with Vertical Slice Representationcited by 8
  20. 2025Occupancy World Model for Robotscited by 8
  21. 2025Self-Supervised Sparse Sensor Fusion for Long Range Perceptioncited by 8
  22. 2025Semantic Causality-Aware Vision-Based 3D Occupancy Predictioncited by 8
  23. 2025OneOcc: Semantic Occupancy Prediction for Legged Robots with a Single Panoramic Cameracited by 8
  24. 2024LightOcc: Lightweight Spatial Embedding for Efficient Vision-based 3D Occupancy Predictioncited by 6
  25. 2024GSRender: Deduplicated Occupancy Prediction via Weakly Supervised 3D Gaussian Splattingcited by 6
  26. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilitiescited by 6
  27. 2025ShelfOcc: Native 3D Supervision beyond LiDAR for Vision-Based Occupancy Estimationcited by 6
  28. 2026OccAny: Generalized Unconstrained Urban 3D Occupancycited by 6
  29. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6
  30. 2026GaussianOcc3D: A Gaussian-Based Adaptive Multi-modal 3D Occupancy Predictioncited by 5
  31. 2026Monocular Open Vocabulary Occupancy Prediction for Indoor Scenescited by 5
  32. 2026VG3S: Visual Geometry Grounded Gaussian Splatting for Semantic Occupancy Predictioncited by 5
  33. 2026Panoramic Multimodal Semantic Occupancy Prediction for Quadruped Robotscited by 5
  34. 2025Inverse++: Vision-Centric 3D Semantic Occupancy Prediction Assisted with 3D Object Detectioncited by 4
  35. 2025ST-GS: Vision-Based 3D Semantic Occupancy Prediction with Spatial-Temporal Gaussian Splattingcited by 4
  36. 2025SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Drivingcited by 4
  37. 2025Progressive Gaussian Transformer with Anisotropy-aware Sampling for Open Vocabulary Occupancy Predictioncited by 4
  38. 2025VG3T: Visual Geometry Grounded Gaussian Transformercited by 4

50 most cited of 157 All 157 (JSON)