mlsanityData to 7 Oct 2026

OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving

arXiv 2405.20337 ↗First posted 30 May 2024cs.CV cs.AIcited by 76

On the map: End-to-end autonomous driving → Generative simulation for driving

Abstract (arXiv)

Understanding the evolution of 3D scenes is important for effective autonomous driving. While conventional methods mode scene development with the motion of individual instances, world models emerge as a generative framework to describe the general scene dynamics. However, most existing methods adopt an autoregressive framework to perform next-token prediction, which suffer from inefficiency in modeling long-term temporal evolutions. To address this, we propose a diffusion-based 4D occupancy generation model, OccSora, to simulate the development of the 3D world for autonomous driving. We employ a 4D scene tokenizer to obtain compact discrete spatial-temporal representations for 4D occupancy input and achieve high-quality reconstruction for long-sequence occupancy videos. We then learn a diffusion transformer on the spatial-temporal representations and generate 4D occupancy conditioned on a trajectory prompt. We conduct extensive experiments on the widely used nuScenes dataset with Occ3D occupancy annotations. OccSora can generate 16s-videos with authentic 3D layout and temporal consistency, demonstrating its ability to understand the spatial and temporal distributions of driving scenes. With trajectory-aware 4D generation, OccSora has the potential to serve as a world simulator for the decision-making of autonomous driving. Code is available at: https://github.com/wzzheng/OccSora.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving. Builds on 59 of its 83 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

59 of 83 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving7 May
  2. 2024GenAD: Generative End-to-End Autonomous Driving18 Feb
  3. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  4. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion2 Nov
  5. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  6. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  8. 2018World Modelscited by 1,444
  9. 2022Planning-oriented Autonomous Drivingcited by 1,068
  10. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  11. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  12. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
38 more
  1. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 325
  2. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 310
  3. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 306
  4. 2022BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervisioncited by 302
  5. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 287
  6. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  7. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  8. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 218
  9. 2023Scene as Occupancycited by 208
  10. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  11. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  12. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  13. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  14. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 143
  15. 2023SelfOcc: Self-Supervised Vision-Based 3D Occupancy Predictioncited by 133
  16. 20223D Object Detection for Autonomous Driving: A Comprehensive Surveycited by 132
  17. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  18. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 116
  19. 2021Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentationcited by 107
  20. 2023BEVHeight: A Robust Framework for Vision-based Roadside 3D Object Detectioncited by 74
  21. 2023DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Drivingcited by 73
  22. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  23. 20223D Object Detection from Images for Autonomous Driving: A Surveycited by 66
  24. 2024SemCity: Semantic Scene Generation with Triplane Diffusioncited by 53
  25. 2023PointOcc: Cylindrical Tri-Perspective View for Point-based 3D Semantic Occupancy Predictioncited by 51
  26. 2023FRNet: Frustum-Range Networks for Scalable LiDAR Segmentationcited by 50
  27. 2023Pyramid Diffusion for Fine 3D Large Scene Generationcited by 44
  28. 2023Camera-based 3D Semantic Scene Completion with Sparse Guidance Networkcited by 40
  29. 2023OccNeRF: Advancing 3D Occupancy Prediction in LiDAR-Free Environmentscited by 39
  30. 2023Flow-Based Feature Fusion for Vehicle-Infrastructure Cooperative 3D Object Detectioncited by 31
  31. 2024OccGen: Generative Multi-modal 3D Occupancy Prediction for Autonomous Drivingcited by 28
  32. 2023Advancing Zero-Shot Digital Human Quality Assessment through Text-Prompted Evaluationcited by 27
  33. 2023LiDAR-based 4D Occupancy Completion and Forecastingcited by 23
  34. 2022Inter-Frame Compression for Dynamic Point Cloud Geometry Codingcited by 21
  35. 2023HeightFormer: Explicit Height Modeling without Extra Data for Camera-only 3D Object Detection in Bird's Eye Viewcited by 19
  36. 2023CoBEV: Elevating Roadside 3D Object Detection with Depth and Height Complementaritycited by 18
  37. 2023BEVGPT: Generative Pre-trained Large Model for Autonomous Driving Prediction, Decision-Making, and Planningcited by 15
  38. 2024OccFusion: Depth Estimation Free Multi-sensor Fusion for 3D Occupancy Predictioncited by 13

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 41; 2026: 19.

76 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  2. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  3. 2024GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Predictioncited by 83
  4. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  5. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  6. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  7. 20253D and 4D World Modeling: A Surveycited by 64
  8. 2025A Survey of World Models for Autonomous Drivingcited by 60
  9. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Modelcited by 58
  10. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  11. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  12. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
38 more
  1. 2024EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understandingcited by 43
  2. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  3. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  4. 2024DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenescited by 37
  5. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  6. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  7. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  8. 2025UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Drivingcited by 27
  9. 2024V2X-Radar: A Multi-modal Dataset with 4D Radar for Cooperative Perceptioncited by 21
  10. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  11. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Drivingcited by 20
  12. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  13. 2025QuadricFormer: Scene as Superquadrics for 3D Semantic Occupancy Predictioncited by 19
  14. 2025TR-DQ: Time-Rotation Diffusion Quantizationcited by 18
  15. 2024Owl-1: Omni World Model for Consistent Long Video Generationcited by 17
  16. 2025GaussRender: Learning 3D Occupancy with Gaussian Renderingcited by 17
  17. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  18. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 15
  19. 2026DriveFuture: Future-Aware Latent World Models for Autonomous Drivingcited by 15
  20. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  21. 2025LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequencescited by 14
  22. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Modelscited by 13
  23. 2025CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Drivingcited by 12
  24. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 11
  25. 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Modelscited by 11
  26. 2025SPIRAL: Semantic-Aware Progressive LiDAR Scene Generation and Understandingcited by 10
  27. 2025Towards foundational LiDAR world models with efficient latent flow matchingcited by 10
  28. 2025GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generationcited by 10
  29. 2025Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Predictioncited by 9
  30. 2025OccTENS: 3D Occupancy World Model via Temporal Next-Scale Predictioncited by 9
  31. 2025Occupancy World Model for Robotscited by 8
  32. 2025I2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecastingcited by 8
  33. 2025Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Methodcited by 8
  34. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  35. 2026SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse Worldcited by 7
  36. 2024Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecastingcited by 6
  37. 2025ShelfOcc: Native 3D Supervision beyond LiDAR for Vision-Based Occupancy Estimationcited by 6
  38. 2024Vision Technologies with Applications in Traffic Surveillance Systems: A Holistic Surveycited by 4

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion-Based Generative Models for 3D Occupancy Prediction in Autonomous Driving
  2. 2024An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Training
  3. 2025OccStress: Stress-Testing the 4D Occupancy Forecasting Chain
  4. 2026OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models
  5. 2023Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications
  6. 2026GEM: Gaussian Evolution Model for Occupancy Forecasting and Motion Planning
  7. 2026InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Driving
  8. 2025Delta-Triplane Transformers as Occupancy World Models