# DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving arXiv: https://arxiv.org/abs/2408.16647 ยท HTML: https://mlsanity.com/paper/2408.16647/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 18 of its 26 references that are in the map. - First posted on arXiv 29 Aug 2024. ## Placed in - Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/) - Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/) - End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/) ## Builds on (most cited first, 18 of 18) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021) - [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018) - [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022) - [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022) - [Flexible Diffusion Modeling of Long Videos](https://mlsanity.com/paper/2205.11495/) (2022) - [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://mlsanity.com/paper/2204.03638/) (2022) - [NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation](https://mlsanity.com/paper/2303.12346/) (2023) - [RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model](https://mlsanity.com/paper/2402.10828/) (2024) - [Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models](https://mlsanity.com/paper/2401.00625/) (2024) - [TrafficFlowGAN: Physics-informed Flow based Generative Adversarial Network for Uncertainty Quantification](https://mlsanity.com/paper/2206.09319/) (2022) - [Quantifying Uncertainty In Traffic State Estimation Using Generative Adversarial Networks](https://mlsanity.com/paper/2206.09349/) (2022) - [Can LLMs Understand Social Norms in Autonomous Driving Games?](https://mlsanity.com/paper/2408.12680/) (2024) - [GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model](https://mlsanity.com/paper/2408.15868/) (2024) ## Built on by (most cited first, 11 of 11) - [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025) - [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025) - [Survey of Video Diffusion Models: Foundations, Implementations, and Applications](https://mlsanity.com/paper/2504.16081/) (2025) - [FashionChameleon: Towards Real-Time and Interactive Human-Garment Video Customization](https://mlsanity.com/paper/2605.15824/) (2026) - [LLM-based Realistic Safety-Critical Driving Video Generation](https://mlsanity.com/paper/2507.01264/) (2025) - [Are AI-Generated Driving Videos Ready for Autonomous Driving? A Diagnostic Evaluation Framework](https://mlsanity.com/paper/2512.06376/) (2025) - [OpenLongTail: Generative Scaling of Long-Tail Driving Data](https://mlsanity.com/paper/2607.09655/) (2026) - [AI-Powered CPS-Enabled Vulnerable-User-Aware Urban Transportation Digital Twin: Methods and Applications](https://mlsanity.com/paper/2501.10396/) (2024) - [Unlock the Power of Unlabeled Data in Language Driving Model](https://mlsanity.com/paper/2503.10586/) (2025) - [A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach](https://mlsanity.com/paper/2512.11944/) (2025) - [Lamarck's Driving School: Discovering Autonomous Driving Training Strategies through Evolutionary Competition](https://mlsanity.com/paper/2610.11662/) (2026) ## Abstract (arXiv) The advancement of autonomous driving technologies necessitates increasingly sophisticated methods for understanding and predicting real-world scenarios. Vision language models (VLMs) are emerging as revolutionary tools with significant potential to influence autonomous driving. In this paper, we propose the DriveGenVLM framework to generate driving videos and use VLMs to understand them. To achieve this, we employ a video generation framework grounded in denoising diffusion probabilistic models (DDPM) aimed at predicting real-world video sequences. We then explore the adequacy of our generated videos for use in VLMs by employing a pre-trained model known as Efficient In-context Learning on Egocentric Videos (EILEV). The diffusion model is trained with the Waymo open dataset and evaluated using the Fr'echet Video Distance (FVD) score to ensure the quality and realism of the generated videos. Corresponding narrations are provided by EILEV for these generated videos, which may be beneficial in the autonomous driving domain. These narrations can enhance traffic scene understanding, aid in navigation, and improve planning capabilities. The integration of video generation with VLMs in the DriveGenVLM framework represents a significant step forward in leveraging advanced AI models to address complex challenges in autonomous driving.