mlsanityData to 14 Oct 2026

From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving

arXiv 2603.17714 ↗First posted 18 Mar 2026cs.AIcited by 0

On the map: End-to-end autonomous driving → Generative simulation for driving

Abstract (arXiv)

Autonomous driving technologies have achieved significant advances in recent years, yet their real-world deployment remains constrained by data scarcity, safety requirements, and the need for generalization across diverse environments. In response, synthetic data and virtual environments have emerged as powerful enablers, offering scalable, controllable, and richly annotated scenarios for training and evaluation. This survey presents a comprehensive review of recent developments at the intersection of autonomous driving, simulation technologies, and synthetic datasets. We organize the landscape across three core dimensions: (i) the use of synthetic data for perception and planning, (ii) digital twin-based simulation for system validation, and (iii) domain adaptation strategies bridging synthetic and real-world data. We also highlight the role of vision-language models and simulation realism in enhancing scene understanding and generalization. A detailed taxonomy of datasets, tools, and simulation platforms is provided, alongside an analysis of trends in benchmark design. Finally, we discuss critical challenges and open research directions, including Sim2Real transfer, scalable safety validation, cooperative autonomy, and simulation-driven policy learning, that must be addressed to accelerate the path toward safe, generalizable, and globally deployable autonomous driving systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving. Builds on 102 of its 193 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

102 of 193 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025RealEngine: Simulating Autonomous Driving in Realistic Context22 May
  2. 2024World Models for Autonomous Driving: An Initial Survey5 Mar
  3. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents8 Feb

Other references · 99

  1. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  5. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  6. 2018YOLOv3: An Incremental Improvementcited by 2,659
  7. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  8. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  9. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  10. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  11. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  12. 2016End to End Learning for Self-Driving Carscited by 1,009
38 more
  1. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  3. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  4. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  5. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  6. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  7. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  8. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  9. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  10. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 403
  11. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
  12. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 360
  13. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 341
  14. 2022V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformercited by 320
  15. 2020V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Predictioncited by 292
  16. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  17. 2018Learning to Drive in a Daycited by 273
  18. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
  19. 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
  20. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 252
  21. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  22. 2018Human Action Recognition and Prediction: A Surveycited by 227
  23. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
  24. 2019High-Resolution Representations for Labeling Pixels and Regionscited by 211
  25. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 207
  26. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 205
  27. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  28. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  29. 2022D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Videocited by 137
  30. 2019Meta-Sim: Learning to Generate Synthetic Datasetscited by 128
  31. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 126
  32. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  33. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
  34. 2020LGSVL Simulator: A High Fidelity Simulator for Autonomous Drivingcited by 121
  35. 2020Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehiclescited by 110
  36. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 106
  37. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 103
  38. 2022COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehiclescited by 95

50 most cited of 99 All 99 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025When Digital Twins Meet Large Language Models: Realistic, Interactive, and Editable Simulation for Autonomous Driving
  2. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
  3. 2022Exiting the Simulation: The Road to Robust and Resilient Autonomous Vehicles at Scale
  4. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Survey
  5. 2025A Survey of World Models for Autonomous Driving
  6. 2025SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integration
  7. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey
  8. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving