mlsanityData to 14 Oct 2026

DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model

arXiv 2410.10738 ↗First posted 14 Oct 2024cs.CV cs.AIcited by 25

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Driving world models have gained increasing attention due to their ability to model complex physical dynamics. However, their superb modeling capability is yet to be fully unleashed due to the limited video diversity in current driving datasets. We introduce DrivingDojo, the first dataset tailor-made for training interactive world models with complex driving dynamics. Our dataset features video clips with a complete set of driving maneuvers, diverse multi-agent interplay, and rich open-world driving knowledge, laying a stepping stone for future world model development. We further define an action instruction following (AIF) benchmark for world models and demonstrate the superiority of the proposed dataset for generating action-controlled future predictions.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 53 of its 69 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

53 of 69 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  2. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
  3. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
  4. 2024World Models for Autonomous Driving: An Initial Survey5 Mar
  5. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)26 Feb
  6. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  7. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  8. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  9. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
  10. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun
  11. 2020One Thousand and One Hours: Self-driving Motion Prediction Dataset25 Jun
  12. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 41

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  7. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  9. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  10. 2018World Modelscited by 1,513
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  12. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
29 more
  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  2. 2023Mastering Diverse Domains through World Modelscited by 1,253
  3. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  4. 2020Mastering Atari with Discrete World Modelscited by 1,007
  5. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  6. 2024Genie: Generative Interactive Environmentscited by 747
  7. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  8. 2022On Distillation of Guided Diffusion Modelscited by 725
  9. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  10. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  11. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 537
  12. 2023Learning Interactive Real-World Simulatorscited by 458
  13. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  14. 2016Generating Videos with Scene Dynamicscited by 376
  15. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 251
  16. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  17. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  18. 2018The ApolloScape Open Dataset for Autonomous Driving and its Applicationcited by 190
  19. 2020A2D2: Audi Autonomous Driving Datasetcited by 190
  20. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 171
  21. 2022Masked World Models for Visual Controlcited by 170
  22. 2022Model-Based Imitation Learning for Urban Drivingcited by 150
  23. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
  24. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  25. 2022Generating Long Videos of Dynamic Scenescited by 113
  26. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  27. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 108
  28. 2024Video as the New Language for Real-World Decision Makingcited by 87
  29. 2023Zenseact Open Dataset: A large-scale and diverse multimodal dataset for autonomous drivingcited by 83

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 9.

25 papers.

Sort:

By area:

  1. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
  2. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  3. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 78
  4. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  5. 20253D and 4D World Modeling: A Surveycited by 67
  6. 2025A Survey of World Models for Autonomous Drivingcited by 60
  7. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  8. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 33
  9. 2026Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Drivingcited by 30
  10. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  11. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  12. 2025AutoData: A Multi-Agent System for Open Web Data Collectioncited by 25
13 more
  1. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  2. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  3. 2025Advancing Off-Road Autonomous Driving: The Large-Scale ORAD-3D Dataset and Comprehensive Benchmarkscited by 8
  4. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
  5. 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectivescited by 5
  6. 2026CameraNoise: Enabling Faithful Camera Control in Video Diffusion through Geometry-Flow-Guided Noise Warpingcited by 4
  7. 2026Motion Forcing: A Decoupled Framework for Robust Video Generation in Motion Dynamicscited by 3
  8. 2026Astronex-World 1.0: Real-Time Interactive World Model Foundationcited by 3
  9. 2025MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformercited by 1
  10. 2026CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generationcited by 1
  11. 2026Driving Video Retrieval for Complex Queries with Structured Groundingcited by 1
  12. 2026Agent-driven Long-tail Simulation for Autonomous Drivingcited by 0
  13. 2026VehDyn: A Driving World Model Benchmark for Vehicle Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos
  2. 2024InfinityDrive: Breaking Time Limits in Driving World Models
  3. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
  4. 2024ACT-Bench: Towards Action Controllable World Models for Autonomous Driving
  5. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models
  6. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation
  7. 2026DriveZero: End-to-End Driving Beyond Human Demonstrations
  8. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving World