mlsanityData to 7 Oct 2026

Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models

arXiv 2506.09042 ↗First posted 10 Jun 2025cs.CVcited by 80

On the map: End-to-end autonomous driving → Generative simulation for driving

Abstract (arXiv)

Collecting and annotating real-world data for safety-critical physical AI systems, such as Autonomous Vehicle (AV), is time-consuming and costly. It is especially challenging to capture rare edge cases, which play a critical role in training and testing of an AV system. To address this challenge, we introduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipeline that aims to generate challenging scenarios to facilitate downstream tasks such as perception and driving policy training. Powering this pipeline is Cosmos-Drive, a suite of models specialized from NVIDIA Cosmos world foundation model for the driving domain and are capable of controllable, high-fidelity, multi-view, and spatiotemporally consistent driving video generation. We showcase the utility of these models by applying Cosmos-Drive-Dreams to scale the quantity and diversity of driving datasets with high-fidelity and challenging scenarios. Experimentally, we demonstrate that our generated data helps in mitigating long-tail distribution problems and enhances generalization in downstream tasks such as 3D lane detection, 3D object detection and driving policy learning. We open source our pipeline toolkit, dataset and model weights through the NVIDIA's Cosmos platform. Project page: https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving. Builds on 58 of its 66 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

58 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving26 Mar
  2. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation17 Oct
  3. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation3 Jun
  4. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  5. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents8 Feb
  6. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  7. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep

Other references · 51

  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  7. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  8. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  9. 2018Albumentations: fast and flexible image augmentationscited by 558
  10. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  11. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  12. 2024YOLOv11: An Overview of the Key Architectural Enhancementscited by 414
38 more
  1. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
  2. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  3. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  4. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  5. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  6. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  7. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 183
  8. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  9. 2025Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Modelscited by 160
  10. 2024OmniRe: Omni Urban Scene Reconstructioncited by 159
  11. 2022PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmarkcited by 140
  12. 20243D Gaussian Ray Tracing: Fast Tracing of Particle Scenescited by 138
  13. 2019Detecting the Unexpected via Image Resynthesiscited by 137
  14. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 132
  15. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  16. 2020LGSVL Simulator: A High Fidelity Simulator for Autonomous Drivingcited by 119
  17. 2016Lost and Found: Detecting Small Road Hazards for Self-Driving Vehiclescited by 117
  18. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  19. 2021Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weathercited by 95
  20. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  21. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 89
  22. 20243DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splattingcited by 87
  23. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  24. 2022LiDAR Snowfall Simulation for Robust 3D Object Detectioncited by 72
  25. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 70
  26. 2023Neural LiDAR Fields for Novel View Synthesiscited by 65
  27. 2024SCube: Instant Large-Scale Scene Reconstruction using VoxSplatscited by 56
  28. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 55
  29. 2024DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Inputcited by 53
  30. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  31. 2023LATR: 3D Lane Detection from Monocular Images with Transformercited by 44
  32. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  33. 2022LET-3D-AP: Longitudinal Error Tolerant 3D Average Precision for Camera-Only 3D Detectioncited by 27
  34. 2023Dynamic LiDAR Re-simulation using Compositional Neural Fieldscited by 22
  35. 2023LightSim: Neural Lighting Simulation for Urban Scenescited by 20
  36. 2022One Ontology to Rule Them All: Corner Case Scenarios for Autonomous Drivingcited by 16
  37. 2021Safety-aware Motion Prediction with Unseen Vehicles for Autonomous Drivingcited by 15
  38. 2023Adv3D: Generating Safety-Critical 3D Objects through Closed-Loop Simulationcited by 10

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 59.

80 papers.

Sort:

By area:

  1. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  2. 2025ViPE: Video Pose Engine for 3D Geometric Perceptioncited by 162
  3. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  4. 2026Advancing Open-source World Modelscited by 144
  5. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  6. 20253D and 4D World Modeling: A Surveycited by 64
  7. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  8. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  9. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  10. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  11. 2025ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulationcited by 28
  12. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 23
38 more
  1. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 22
  2. 2026MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokenscited by 20
  3. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  4. 2026NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulationcited by 14
  5. 2026Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representationcited by 13
  6. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 13
  7. 2026LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generationcited by 12
  8. 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Modelscited by 11
  9. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  10. 2026Gamma-World: Generative Multi-Agent World Modeling Beyond Two Playerscited by 10
  11. 2025Recurrent Autoregressive Diffusion: Global Memory Meets Local Attentioncited by 9
  12. 2026UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Drivingcited by 9
  13. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  14. 2026MAD: Motion Appearance Decoupling for efficient Driving World Modelscited by 7
  15. 2026Geometry-Aware Implicit Memory for Video World Modelscited by 7
  16. 2025TeraSim-World: Worldwide Safety-Critical Data Synthesis for End-to-End Autonomous Drivingcited by 6
  17. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  18. 2026Learning to Drive in New Cities Without Human Demonstrationscited by 6
  19. 2026RoboTrustBench: Benchmarking the Trustworthiness of Video World Models for Robotic Manipulationcited by 6
  20. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
  21. 2026Human Cognition in Machines: A Unified Perspective of World Modelscited by 5
  22. 2026IT-OSE: Exploring Optimal Sample Size for Industrial Data Augmentationcited by 4
  23. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  24. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  25. 2026HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehiclescited by 3
  26. 2026HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulationcited by 3
  27. 2026Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Modelcited by 3
  28. 2025RadarGen: Automotive Radar Point Cloud Generation from Camerascited by 2
  29. 2026Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Drivingcited by 2
  30. 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Controlcited by 2
  31. 2026DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Modelscited by 2
  32. 2026MoWorld: A Flash World Modelcited by 2
  33. 2026Cyclone: Diffusion Model for Cycle-Consistent Weather Editing from Unpaired Driving Datacited by 2
  34. 2026Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulationcited by 2
  35. 2026Pictura: Perspective-View Self-Play at Scale for Drivingcited by 2
  36. 2025CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal Reconstruction Model for Autonomous Drivingcited by 1
  37. 2025Visionary: The World Model Carrier Built on WebGPU-Powered Gaussian Splatting Platformcited by 1
  38. 2026LiDAR-EVS: Enhance Extrapolated View Synthesis for 3D Gaussian Splatting with Pseudo-LiDAR Supervisioncited by 1

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
  2. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
  3. 2024InfinityDrive: Breaking Time Limits in Driving World Models
  4. 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
  5. 2024DreamDrive: Generative 4D Scene Modeling from Street View Images
  6. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities
  7. 2024GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model
  8. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving