mlsanityData to 14 Oct 2026

GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction

arXiv 2412.10373 ↗First posted 13 Dec 2024cs.CV cs.AI cs.LGcited by 67

On the map: World models → World models for autonomous driving

Abstract (arXiv)

3D occupancy prediction is important for autonomous driving due to its comprehensive perception of the surroundings. To incorporate sequential inputs, most existing methods fuse representations from previous frames to infer the current 3D occupancy. However, they fail to consider the continuity of driving scenarios and ignore the strong prior provided by the evolution of 3D scenes (e.g., only dynamic objects move). In this paper, we propose a world-model-based framework to exploit the scene evolution for perception. We reformulate 3D occupancy prediction as a 4D occupancy forecasting problem conditioned on the current sensor input. We decompose the scene evolution into three factors: 1) ego motion alignment of static scenes; 2) local movements of dynamic objects; and 3) completion of newly-observed scenes. We then employ a Gaussian world model (GaussianWorld) to explicitly exploit these priors and infer the scene evolution in the 3D Gaussian space considering the current RGB observation. We evaluate the effectiveness of our framework on the widely used nuScenes dataset. Our GaussianWorld improves the performance of the single-frame counterpart by over 2% in mIoU without introducing additional computations. Code: https://github.com/zuosc19/GaussianWorld.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 51 of its 56 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

51 of 56 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation17 Oct
  2. 2024Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models25 Sep
  3. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving5 Sep
  4. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving26 Aug
  5. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  6. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving7 May
  7. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
  8. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Driving29 Dec
  9. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  10. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  11. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  12. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
3 more
  1. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving27 Apr
  2. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting25 Feb
  3. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers31 Mar

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  6. 2018World Modelscited by 1,513
  7. 2022Planning-oriented Autonomous Drivingcited by 1,087
  8. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 713
  9. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  10. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 572
  11. 2021FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detectioncited by 447
  12. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
24 more
  1. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 363
  2. 2022BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detectioncited by 341
  3. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 316
  4. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 315
  5. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
  6. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 293
  7. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 252
  8. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 223
  9. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 221
  10. 2023Scene as Occupancycited by 209
  11. 2020Atlas: End-to-End 3D Scene Reconstruction from Posed Imagescited by 206
  12. 2020LMSCNet: Lightweight Multiscale 3D Semantic Completioncited by 188
  13. 2023FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformationcited by 165
  14. 2022Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusioncited by 162
  15. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 157
  16. 2022Model-Based Imitation Learning for Urban Drivingcited by 150
  17. 2023PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentationcited by 115
  18. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  19. 2023COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Predictioncited by 96
  20. 2023Symphonize 3D Semantic Scene Completion with Contextual Instance Queriescited by 90
  21. 2023PointOcc: Cylindrical Tri-Perspective View for Point-based 3D Semantic Occupancy Predictioncited by 52
  22. 2023OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queriescited by 48
  23. 2024Occupancy as Set of Pointscited by 38
  24. 2024CVT-Occ: Cost Volume Temporal Fusion for 3D Occupancy Predictioncited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 35; 2026: 31.

67 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  2. 20253D and 4D World Modeling: A Surveycited by 67
  3. 2025A Survey of World Models for Autonomous Drivingcited by 60
  4. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  5. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  6. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 39
  7. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 38
  8. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  9. 2025GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flowcited by 24
  10. 2025DVGT: Driving Visual Geometry Transformercited by 21
  11. 2025QuadricFormer: Scene as Superquadrics for 3D Semantic Occupancy Predictioncited by 20
  12. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
38 more
  1. 2025LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequencescited by 17
  2. 2025GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attentioncited by 14
  3. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 13
  4. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  5. 2025Four Principles for Physically Interpretable World Modelscited by 10
  6. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  7. 2025Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Predictioncited by 9
  8. 2025OneOcc: Semantic Occupancy Prediction for Legged Robots with a Single Panoramic Cameracited by 8
  9. 2026MAD: Motion Appearance Decoupling for efficient Driving World Modelscited by 8
  10. 2025RoDyn: Taming Interactive Robot-Dynamic 2.5D World Model for Robotic Manipulationcited by 7
  11. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6
  12. 2025EvolvingAgent: Curriculum Self-evolving Agent with Continual World Model for Long-Horizon Taskscited by 5
  13. 2026A Definition and Roadmap for World Modelscited by 5
  14. 2025ST-GS: Vision-Based 3D Semantic Occupancy Prediction with Spatial-Temporal Gaussian Splattingcited by 4
  15. 2025SQS: Enhancing Sparse Perception Models via Query-based Splatting in Autonomous Drivingcited by 4
  16. 2026RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Spacecited by 4
  17. 2026CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluationcited by 4
  18. 2024Physically Interpretable World Models via Weakly Supervised Representation Learningcited by 3
  19. 2025RadarGaussianDet3D: Gaussian Representation-based Real-time 3D Object Detection with 4D Automotive Radarscited by 3
  20. 2025ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understandingcited by 3
  21. 2025GraphGSOcc: Semantic-Geometric Graph Transformer with Dynamic-Static Decoupling for 3D Gaussian Splatting-based Occupancy Predictioncited by 2
  22. 2025Learning to Generate 4D LiDAR Sequencescited by 2
  23. 2025X-WIN: Building Chest Radiograph World Model via Predictive Sensingcited by 2
  24. 2025Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehiclescited by 2
  25. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  26. 2026DLWM: Dual Latent World Models enable Holistic Gaussian-centric Pre-training in Autonomous Drivingcited by 2
  27. 2026SparseWorld: Enhancing End-to-End Autonomous Driving via World Models with Sparse Scene Representationcited by 2
  28. 2025S2GO: Streaming Sparse Gaussian Occupancy Predictioncited by 1
  29. 2025Maps for Autonomous Driving: Full-process Survey and Frontierscited by 1
  30. 2026SuperOcc: Toward Cohesive Temporal Modeling for Superquadric-based 3D Occupancy Predictioncited by 1
  31. 2026Risk-Aware World Model Predictive Control for Generalizable End-to-End Autonomous Drivingcited by 1
  32. 2026Gau-Occ: Geometry-Completed Gaussians for Multi-Modal 3D Occupancy Predictioncited by 1
  33. 2026VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Predictioncited by 1
  34. 2026SurroundNEXO: Ego-Centric Metric Bridging for Spatially Consistent Geometry in Autonomous Drivingcited by 1
  35. 2026FLM-Occ: Feed-forward Likelihood Maximization for Efficient Indoor Occupancy Predictioncited by 1
  36. 2025Streaming Dense Voxel Representations for 3D Occupancy Predictioncited by 0
  37. 2025LiSTAR: Ray-Centric World Models for 4D LiDAR Sequences in Autonomous Drivingcited by 0
  38. 2025SuperQuadricOcc: Real-Time Self-Supervised Semantic Occupancy Estimation with Superquadric Volume Renderingcited by 0

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction
  2. 2023Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications
  3. 2025Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving
  4. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving
  5. 2025Diffusion-Based Generative Models for 3D Occupancy Prediction in Autonomous Driving
  6. 2026GaussianOcc3D: A Gaussian-Based Adaptive Multi-modal 3D Occupancy Prediction
  7. 2025OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction
  8. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Training