mlsanityData to 14 Oct 2026

UniScene: Unified Occupancy-centric Driving Scene Generation

arXiv 2412.05435 ↗First posted 6 Dec 2024cs.CVcited by 106

Not on the map. Why?

Cited by 46 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating high-fidelity, controllable, and annotated training data is critical for autonomous driving. Existing methods typically generate a single data form directly from a coarse scene layout, which not only fails to output rich data forms required for diverse downstream tasks but also struggles to model the direct layout-to-data distribution. In this paper, we introduce UniScene, the first unified framework for generating three key data forms - semantic occupancy, video, and LiDAR - in driving scenes. UniScene employs a progressive generation process that decomposes the complex task of scene generation into two hierarchical steps: (a) first generating semantic occupancy from a customized scene layout as a meta scene representation rich in both semantic and geometric information, and then (b) conditioned on occupancy, generating video and LiDAR data, respectively, with two novel transfer strategies of Gaussian-based Joint Rendering and Prior-guided Sparse Modeling. This occupancy-centric approach reduces the generation burden, especially for intricate scenes, while providing detailed intermediate representations for the subsequent generation stages. Extensive experiments demonstrate that UniScene outperforms previous SOTAs in the occupancy, video, and LiDAR generation, which also indeed benefits downstream driving tasks. Project page: https://arlo0o.github.io/uniscene/

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Builds on 75 of its 86 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

75 of 86 references.

References · 75

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  9. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  11. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  12. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
38 more
  1. 2022Planning-oriented Autonomous Drivingcited by 1,087
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  3. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  4. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 735
  5. 2018Open3D: A Modern Library for 3D Data Processingcited by 721
  6. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 671
  7. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  8. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  9. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 409
  10. 2019From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Networkcited by 368
  11. 2021MonoScene: Monocular 3D Semantic Scene Completioncited by 363
  12. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  13. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 336
  14. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  15. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  16. 2023SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Drivingcited by 316
  17. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  18. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  19. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  20. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 252
  21. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 233
  22. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  23. 2020Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completioncited by 223
  24. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 221
  25. 2023Scene as Occupancycited by 209
  26. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 191
  27. 2021The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depthcited by 190
  28. 2020LMSCNet: Lightweight Multiscale 3D Semantic Completioncited by 188
  29. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  30. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 168
  31. 2020Controllable Person Image Synthesis with Attribute-Decomposed GANcited by 150
  32. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 142
  33. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 127
  34. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 126
  35. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  36. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 123
  37. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  38. 2018GAN Augmentation: Augmenting Training Data using Generative Adversarial Networkscited by 107

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 60; 2026: 46.

106 papers.

Sort:

By area:

  1. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  2. 20253D and 4D World Modeling: A Surveycited by 67
  3. 2025A Survey of World Models for Autonomous Drivingcited by 60
  4. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  5. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 47
  6. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 45
  7. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 40
  8. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 36
  9. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 33
  10. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 33
  11. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  12. 20253D Scene Generation: A Surveycited by 24
38 more
  1. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  2. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 22
  3. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  4. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 19
  5. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  6. 2025LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequencescited by 17
  7. 2025Challenger: Affordable Adversarial Driving Video Generationcited by 16
  8. 2025WorldGrow: Generating Infinite 3D Worldcited by 16
  9. 2025Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splattingcited by 15
  10. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 15
  11. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  12. 2025COME: Adding Scene-Centric Forecasting Control to Occupancy World Modelcited by 14
  13. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Taskscited by 14
  14. 2025MapDiffusion: Generative Diffusion for Vectorized Online HD Map Construction and Uncertainty Estimation in Autonomous Drivingcited by 13
  15. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 13
  16. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  17. 2025CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Drivingcited by 12
  18. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 12
  19. 2026CoWorld-VLA: Thinking in a Multi-Expert World Model for Autonomous Drivingcited by 12
  20. 2025ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidancecited by 11
  21. 2025CRUISE: Cooperative Reconstruction and Editing in V2X Scenarios using Gaussian Splattingcited by 11
  22. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11
  23. 2025GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generationcited by 11
  24. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  25. 2025SPIRAL: Semantic-Aware Progressive LiDAR Scene Generation and Understandingcited by 10
  26. 2025Towards foundational LiDAR world models with efficient latent flow matchingcited by 10
  27. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  28. 2025Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimationcited by 10
  29. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  30. 2025LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Modelcited by 9
  31. 2025OccTENS: 3D Occupancy World Model via Temporal Next-Scale Predictioncited by 9
  32. 2026UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Drivingcited by 9
  33. 2025MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstructioncited by 8
  34. 2025I2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecastingcited by 8
  35. 20254D Driving Scene Generation With Stereo Forcingcited by 8
  36. 2025Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Methodcited by 8
  37. 2026ReWorld: Multi-Dimensional Reward Modeling for Embodied World Modelscited by 8
  38. 2026World Engine: Towards the Era of Post-Training for Autonomous Drivingcited by 8

50 most cited of 106 All 106 (JSON)