Generative AI for Autonomous Driving: Frontiers and Opportunities
On the map: End-to-end autonomous driving → Generative simulation for driving
Abstract (arXiv)
Generative Artificial Intelligence (GenAI) constitutes a transformative technological wave that reconfigures industries through its unparalleled capabilities for content creation, reasoning, planning, and multimodal understanding. This revolutionary force offers the most promising path yet toward solving one of engineering's grandest challenges: achieving reliable, fully autonomous driving, particularly the pursuit of Level 5 autonomy. This survey delivers a comprehensive and critical synthesis of the emerging role of GenAI across the autonomous driving stack. We begin by distilling the principles and trade-offs of modern generative modeling, encompassing VAEs, GANs, Diffusion Models, and Large Language Models (LLMs). We then map their frontier applications in image, LiDAR, trajectory, occupancy, video generation as well as LLM-guided reasoning and decision making. We categorize practical applications, such as synthetic data workflows, end-to-end driving strategies, high-fidelity digital twin systems, smart transportation networks, and cross-domain transfer to embodied AI. We identify key obstacles and possibilities such as comprehensive generalization across rare cases, evaluation and safety checks, budget-limited implementation, regulatory compliance, ethical concerns, and environmental effects, while proposing research plans across theoretical assurances, trust metrics, transport integration, and socio-technical influence. By unifying these threads, the survey provides a forward-looking reference for researchers, engineers, and policymakers navigating the convergence of generative AI and advanced autonomous mobility. An actively maintained repository of cited works is available at https://github.com/taco-group/GenAI4AD.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving. Builds on 235 of its 912 references that are in the map. First posted on arXiv 13 May 2025.
Builds on
235 of 912 references.
Closest ancestors · 17
Same direction, newest first
- 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments28 Mar
- 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving26 Mar
- 2025CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Driving9 Mar
- 2025A Survey of World Models for Autonomous Driving20 Jan
- 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT27 Dec
- 2024Doe-1: Closed-Loop Autonomous Driving with Large World Model12 Dec
- 2024Driving View Synthesis on Free-form Trajectories with Generative Prior2 Dec
- 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving2 Dec
- 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey5 Nov
- 2024DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation9 Sep
- 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving1 Aug
- 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation3 Jun
5 more
- 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving30 May
- 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
- 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
- 2024GenAD: Generative End-to-End Autonomous Driving18 Feb
- 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep
Other references · 218
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023GPT-4 Technical Reportcited by 18,396
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2022Classifier-Free Diffusion Guidancecited by 5,847
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
38 more
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2024Qwen2 Technical Reportcited by 2,120
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
- 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
- 2018Demystifying MMD GANscited by 1,293
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
- 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
- 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
- 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
- 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
- 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
- 2017Wasserstein GANcited by 639
- 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
- 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
- 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
- 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
- 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
- 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
- 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
- 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
- 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
- 2023Learning Interactive Real-World Simulatorscited by 458
- 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
- 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 420
- 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
- 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
- 2023Llemma: An Open Language Model For Mathematicscited by 388
- 2023GPT-Driver: Learning to Drive with GPTcited by 347
- 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 334
- 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
50 most cited of 218 All 218 (JSON)
Built on by
Citing papers per year: 2025: 6; 2026: 1.
7 papers.
- 20253D and 4D World Modeling: A Surveycited by 67
- 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 23
- 2025DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinkingcited by 12
- 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
- 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectivescited by 5
- 2025DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Drivingcited by 5
- 2026Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Supportcited by 0
- 2026Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Support31 May 2026
- 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World11 Dec 2025
- 2025A Survey of Inductive Reasoning for Large Language Models11 Oct 2025
- 20253D and 4D World Modeling: A Survey4 Sep 2025
- 2025DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking28 Jul 2025
- 2025DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving21 Jun 2025
- 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives14 May 2025
Similar, not cited
Close in wording, no citation either way
- 2025Generative AI for Autonomous Driving: A Review
- 2023GAIA-1: A Generative World Model for Autonomous Driving
- 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
- 2026Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges
- 2024Exploring Generative AI for Sim2Real in Driving Data Synthesis
- 2021Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions
- 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
- 2024GPD-1: Generative Pre-training for Driving