mlsanityData to 14 Oct 2026

Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis

arXiv 2506.11526 ↗First posted 13 Jun 2025cs.RO cs.AIcited by 38

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

For autonomous vehicles, safe navigation in complex environments depends on handling a broad range of diverse and rare driving scenarios. Simulation- and scenario-based testing have emerged as key approaches to development and validation of autonomous driving systems. Traditional scenario generation relies on rule-based systems, knowledge-driven models, and data-driven synthesis, often producing limited diversity and unrealistic safety-critical cases. With the emergence of foundation models, which represent a new generation of pre-trained, general-purpose AI models, developers can process heterogeneous inputs (e.g., natural language, sensor data, HD maps, and control actions), enabling the synthesis and interpretation of complex driving scenarios. In this paper, we conduct a survey about the application of foundation models for scenario generation and scenario analysis in autonomous driving (as of May 2025). Our survey presents a unified taxonomy that includes large language models, vision-language models, multimodal large language models, diffusion models, and world models for the generation and analysis of autonomous driving scenarios. In addition, we review the methodologies, open-source datasets, simulation platforms, and benchmark challenges, and we examine the evaluation metrics tailored explicitly to scenario generation and analysis. Finally, the survey concludes by highlighting the open challenges and research questions, and outlining promising future research directions. All reviewed papers are listed in a continuously maintained repository, which contains supplementary materials and is available at https://github.com/TUM-AVS/FM-for-Scenario-Generation-Analysis.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 255 of its 358 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

255 of 358 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarios16 May
  2. 2025Seeking to Collide: Online Safety-Critical Scenario Generation for Autonomous Driving with Retrieval Augmented Large Language Models2 May
  3. 2025Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test4 Mar
  4. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances24 Feb
  5. 2025SenseRAG: Constructing Environmental Knowledge Bases with Proactive Querying for LLM-Based Autonomous Driving7 Jan
  6. 2024Realistic Corner Case Generation for Autonomous Vehicles with Multimodal Large Language Model29 Nov
  7. 2024A Survey on Large Language Model-empowered Autonomous Driving21 Sep
  8. 2024XLM for Autonomous Driving Systems: A Comprehensive Review16 Sep
  9. 2024Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model15 Sep
  10. 2024Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles10 Sep
  11. 2024Large Language Models for Human-like Autonomous Driving: A Survey27 Jul
  12. 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles22 May
7 more
  1. 2024Large Language Models Powered Context-aware Motion Prediction in Autonomous Driving17 Mar
  2. 2024A Survey for Foundation Models in Autonomous Driving2 Feb
  3. 2023A Survey on Multimodal Large Language Models for Autonomous Driving21 Nov
  4. 2023Applications of Large Scale Foundation Models for Autonomous Driving20 Nov
  5. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving2 Nov
  6. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  7. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 236

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2023GPT-4 Technical Reportcited by 18,396
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  11. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  12. 2020Denoising Diffusion Implicit Modelscited by 9,781
38 more
  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  13. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  14. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  15. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  16. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  17. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  18. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  19. 2023Qwen Technical Reportcited by 3,344
  20. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  21. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  22. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  23. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  24. 2022Video Diffusion Modelscited by 2,305
  25. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  26. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  27. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  28. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  29. 2018World Modelscited by 1,513
  30. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  31. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  32. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  33. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  34. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  35. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  36. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  37. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  38. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785

50 most cited of 236 All 236 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 27.

38 papers.

Sort:

By area:

  1. 2025Evaluating Gemini Robotics Policies in a Veo World Simulatorcited by 51
  2. 2025BAgger: Backwards Aggregation for Mitigating Drift in Autoregressive Video Diffusion Modelscited by 16
  3. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  4. 2026α3-Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents over 6G Networkscited by 10
  5. 2025NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Drivingcited by 6
  6. 2026StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Drivingcited by 6
  7. 2026Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Drivingcited by 5
  8. 2026AgentDrive: An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systemscited by 4
  9. 2026StreamingClaw Technical Reportcited by 4
  10. 2025Controllable Collision Scenario Generation via Collision Pattern Predictioncited by 3
  11. 2026Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Drivingcited by 3
  12. 2025Target-Bench: Can Video World Models Achieve Mapless Path Planning with Semantic Targets?cited by 2
26 more
  1. 2026How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Studycited by 2
  2. 2025A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systemscited by 1
  3. 2025Learning from Risk: LLM-Guided Generation of Safety-Critical Scenarios with Prior Knowledgecited by 1
  4. 2026AnchorDrive: LLM Scenario Rollout with Anchor-Guided Diffusion Regeneration for Safety-Critical Scenario Generationcited by 1
  5. 2026RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operationcited by 1
  6. 2026FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Modelcited by 1
  7. 2026World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planningcited by 1
  8. 2026Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Drivingcited by 1
  9. 2025Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoningcited by 0
  10. 2025Driving scenario generation and evaluation using a structured layer representation and foundational modelscited by 0
  11. 2025OpenTwinMap: An Open-Source Digital Twin Generator for Urban Autonomous Drivingcited by 0
  12. 2026SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Finecited by 0
  13. 2026Supporting software engineering tasks with agentic AI: Demonstration on document retrieval and test scenario generationcited by 0
  14. 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Drivingcited by 0
  15. 2026TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarioscited by 0
  16. 2026Artificial Intelligence for Modeling and Simulation of Mixed Automated and Human Trafficcited by 0
  17. 2026Beyond Known Objects: A Novel Framework for Open-Set Object Detection using Negative-Aware Normcited by 0
  18. 2026Operating Within the Operational Design Domain: Zero-Shot Perception with Vision-Language Modelscited by 0
  19. 2026GPS-Enhanced Tourist Mobility Modeling with Seasonal Spatial Priors and LLM-Based Activity Chain Generationcited by 0
  20. 2026Search-based Testing of Vision Language Models for In-Car Scene Understandingcited by 0
  21. 2026End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generationcited by 0
  22. 2026WM-Cov: Test Adequacy for Interactive World-Model-Style Autonomous Driving Simulationcited by 0
  23. 2026GSRAIN: Physically Calibrated High-/Low-Frequency Rainfall Synthesis for 3D Gaussian Driving Scenescited by 0
  24. 2026UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edgecited by 0
  25. 2026LLM-Guided Transformation of Non-Critical Driving Scenes into Safety-Critical Scenarios Using Augmented Realitycited by 0
  26. 2026Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistantscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
  2. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
  3. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities
  4. 2025VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion
  5. 2022Exiting the Simulation: The Road to Robust and Resilient Autonomous Vehicles at Scale
  6. 2025A Tool for Benchmarking Large Language Models' Robustness in Assessing the Realism of Driving Scenarios ↗
  7. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey
  8. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving