mlsanityData to 14 Oct 2026

GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention

arXiv 2505.10685 ↗First posted 15 May 2025cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

3D semantic occupancy prediction is essential for achieving safe, reliable autonomous driving and robotic navigation. Compared to camera-only perception systems, multi-modal pipelines, especially LiDAR-camera fusion methods, can produce more accurate and fine-grained predictions. Although voxel-based scene representations are widely used for semantic occupancy prediction, 3D Gaussians have emerged as a continuous and significantly more compact alternative. In this work, we propose a multi-modal Gaussian-based semantic occupancy prediction framework utilizing 3D deformable attention, namely GaussianFormer3D. We introduce a voxel-to-Gaussian initialization strategy that provides 3D Gaussians with accurate geometry priors from LiDAR data, and design a LiDAR-guided 3D deformable attention mechanism to refine these Gaussians using LiDAR-camera fusion features in a lifted 3D space. Extensive experiments on real-world on-road and off-road autonomous driving datasets demonstrate that GaussianFormer3D achieves state-of-the-art prediction performance with reduced memory consumption and improved efficiency.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 89 of its 97 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

89 of 97 references.

References · 89

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  11. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  12. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
38 more
  1. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  2. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  3. 2022Planning-oriented Autonomous Drivingcited by 1,087
  4. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  5. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 941
  6. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
  7. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  8. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  9. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 572
  10. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
  11. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 447
  12. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
  13. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 363
  14. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 360
  15. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 341
  16. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 332
  17. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 316
  18. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
  19. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  20. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 293
  21. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 223
  22. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 221
  23. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splattingcited by 221
  24. 2020LMSCNet: Lightweight Multiscale 3D Semantic Completioncited by 188
  25. 2023FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformationcited by 165
  26. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 157
  27. 2023SelfOcc: Self-Supervised Vision-Based 3D Occupancy Predictioncited by 133
  28. 2023RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervisioncited by 128
  29. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 127
  30. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 126
  31. 2020RELLIS-3D Dataset: Data, Benchmarks and Analysiscited by 119
  32. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 115
  33. 2023FlashOcc: Fast and Memory-Efficient Occupancy Prediction via Channel-to-Height Plugincited by 103
  34. 2020S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Cloudscited by 102
  35. 2018In Defense of Classical Image Processing: Fast Depth Completion on the CPUcited by 96
  36. 2023COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Predictioncited by 96
  37. 2023Symphonize 3D Semantic Scene Completion with Contextual Instance Queriescited by 90
  38. 2023OccDepth: A Depth-Aware Method for 3D Semantic Scene Completioncited by 86

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025SplatSSC: Decoupled Depth-Guided Gaussian Splatting for Semantic Scene Completioncited by 9
  2. 2026GaussianOcc3D: A Gaussian-Based Adaptive Multi-modal 3D Occupancy Predictioncited by 5
  3. 2025Vision-Only Gaussian Splatting for Collaborative Semantic Occupancy Predictioncited by 3
  4. 2025ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understandingcited by 3
  5. 2025Easy3D-Labels: Supervising Semantic Occupancy Estimation with 3D Pseudo-Labels for Automotive Perceptioncited by 2
  6. 2025TGSFormer: Scalable Temporal Gaussian Splatting for Embodied Semantic Scene Completioncited by 2
  7. 2026Gau-Occ: Geometry-Completed Gaussians for Multi-Modal 3D Occupancy Predictioncited by 1
  8. 2026VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Predictioncited by 1
  9. 2025OccLE: Label-Efficient 3D Semantic Occupancy Predictioncited by 0
  10. 2025SuperQuadricOcc: Real-Time Self-Supervised Semantic Occupancy Estimation with Superquadric Volume Renderingcited by 0
  11. 2026TFusionOcc: T-Primitive Based Object-Centric Multi-Sensor Fusion Framework for 3D Occupancy Predictioncited by 0
  12. 2026ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Predictioncited by 0
2 more
  1. 2026GaussianSeed: Hierarchical Gaussian Seeding for High-Resolution 3D Occupancy Predictioncited by 0
  2. 2026Learning Adaptive Semantic Gaussian Allocation for 3D Occupancycited by 0