mlsanityData to 7 Oct 2026

OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving

arXiv 2311.16038 ↗First posted 27 Nov 2023cs.CV cs.AI cs.LGcited by 230

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Understanding how the 3D scene evolves is vital for making decisions in autonomous driving. Most existing methods achieve this by predicting the movements of object boxes, which cannot capture more fine-grained scene information. In this paper, we explore a new framework of learning a world model, OccWorld, in the 3D Occupancy space to simultaneously predict the movement of the ego car and the evolution of the surrounding scenes. We propose to learn a world model based on 3D occupancy rather than 3D bounding boxes and segmentation maps for three reasons: 1) expressiveness. 3D occupancy can describe the more fine-grained 3D structure of the scene; 2) efficiency. 3D occupancy is more economical to obtain (e.g., from sparse LiDAR points). 3) versatility. 3D occupancy can adapt to both vision and LiDAR. To facilitate the modeling of the world evolution, we learn a reconstruction-based scene tokenizer on the 3D occupancy to obtain discrete scene tokens to describe the surrounding scenes. We then adopt a GPT-like spatial-temporal generative transformer to generate subsequent scene and ego tokens to decode the future occupancy and ego trajectory. Extensive experiments on the widely used nuScenes benchmark demonstrate the ability of OccWorld to effectively model the evolution of the driving scenes. OccWorld also produces competitive planning results without using instance and map supervision. Code: https://github.com/wzzheng/OccWorld.

Key facts. Cited by 230 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 61 of its 68 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

61 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  2. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  3. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving27 Apr
  4. 2023Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting25 Feb
  5. 2022Differentiable Raycasting for Self-supervised Occupancy Forecasting4 Oct
  6. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers31 Mar

Other references · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2013Auto-Encoding Variational Bayescited by 3,980
  11. 2018World Modelscited by 1,444
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
38 more
  1. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  2. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  3. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  4. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  5. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 422
  6. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  7. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 388
  8. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  9. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 357
  10. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 326
  11. 2022MapTR: Structured Modeling and Learning for Online Vectorized HD Map Constructioncited by 311
  12. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  13. 2021HDMapNet: An Online HD Map Construction and Evaluation Frameworkcited by 280
  14. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  15. 2022VectorMapNet: End-to-end Vectorized HD Map Learningcited by 243
  16. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  17. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  18. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 222
  19. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 218
  20. 2023Scene as Occupancycited by 208
  21. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  22. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  23. 2020LMSCNet: Lightweight Multiscale 3D Semantic Completioncited by 187
  24. 2021(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Networkcited by 164
  25. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  26. 2023SelfOcc: Self-Supervised Vision-Based 3D Occupancy Predictioncited by 133
  27. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 126
  28. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 124
  29. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  30. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  31. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  32. 2020AMVNet: Assertion-based Multi-View Fusion Network for LiDAR Semantic Segmentationcited by 87
  33. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  34. 2022LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perceptioncited by 74
  35. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  36. 2021SafetyNet: Safe planning for real-world self-driving vehicles using machine-learned policiescited by 68
  37. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  38. 2023PointOcc: Cylindrical Tri-Perspective View for Point-based 3D Semantic Occupancy Predictioncited by 51

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 45; 2025: 96; 2026: 87.

230 papers.

Sort:

By area:

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  2. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  3. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  4. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  5. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  6. 2024GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Predictioncited by 155
  7. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  8. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  9. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  10. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  11. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Modelcited by 91
  12. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
38 more
  1. 2024GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Predictioncited by 83
  2. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  3. 2025Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memorycited by 82
  4. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  5. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  6. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  7. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  8. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  9. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPTcited by 66
  10. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  11. 20253D and 4D World Modeling: A Surveycited by 64
  12. 2025A Survey of World Models for Autonomous Drivingcited by 60
  13. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Modelcited by 58
  14. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  15. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  16. 2024Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Drivingcited by 46
  17. 2024RenderWorld: World Model with Self-Supervised 3D Labelcited by 44
  18. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 43
  19. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  20. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  21. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  22. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  23. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  24. 2024A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspectivecited by 37
  25. 2024DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenescited by 37
  26. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  27. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  28. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  29. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  30. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  31. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  32. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  33. 2024Vision-based 3D occupancy prediction in autonomous driving: a review and outlookcited by 29
  34. 2025Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Drivingcited by 29
  35. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  36. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  37. 2025Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Modelscited by 27
  38. 2025UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Drivingcited by 27

50 most cited of 230 All 230 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion-Based Generative Models for 3D Occupancy Prediction in Autonomous Driving
  2. 2024OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Driving
  3. 2025OccLE: Label-Efficient 3D Semantic Occupancy Prediction
  4. 2026Dr.Occ: Depth- and Region-Guided 3D Occupancy from Surround-View Cameras for Autonomous Driving
  5. 2023UniWorld: Autonomous Driving Pre-training via World Models
  6. 2024UnO: Unsupervised Occupancy Fields for Perception and Forecasting
  7. 2023Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications
  8. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations