# LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving arXiv: https://arxiv.org/abs/2505.00284 ยท HTML: https://mlsanity.com/paper/2505.00284/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 32 of its 42 references that are in the map. - First posted on arXiv 1 May 2025. ## Placed in - Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/) - End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/) ## Builds on (most cited first, 30 of 32) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017) - [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022) - [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022) - [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019) - [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023) - [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023) - [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023) - [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024) - [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023) - [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022) - [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023) - [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021) - [Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?](https://mlsanity.com/paper/2312.03031/) (2023) - [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023) - [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024) - [Textual Explanations for Self-Driving Vehicles](https://mlsanity.com/paper/1807.11546/) (2018) - [Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes](https://mlsanity.com/paper/2305.10430/) (2023) - [Distilling Task-Specific Knowledge from BERT into Simple Neural Networks](https://mlsanity.com/paper/1903.12136/) (2019) - [End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances](https://mlsanity.com/paper/1911.10868/) (2019) - [Dolphins: Multimodal Language Model for Driving](https://mlsanity.com/paper/2312.00438/) (2023) - [Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving](https://mlsanity.com/paper/2312.03661/) (2023) - [Language Prompt for Autonomous Driving](https://mlsanity.com/paper/2309.04379/) (2023) - [InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language](https://mlsanity.com/paper/2305.05662/) (2023) - [OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2412.15208/) (2024) - ["Curse of rarity" for autonomous vehicles](https://mlsanity.com/paper/2207.02749/) (2022) ## Built on by (most cited first, 6 of 6) - [dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning](https://mlsanity.com/paper/2512.04459/) (2025) - [Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects](https://mlsanity.com/paper/2505.20223/) (2025) - [DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking](https://mlsanity.com/paper/2507.20879/) (2025) - [Less is More: Lean yet Powerful Vision-Language Model for Autonomous Driving](https://mlsanity.com/paper/2510.00060/) (2025) - [Reasoning About Traversability: Language-Guided Off-Road 3D Trajectory Planning](https://mlsanity.com/paper/2604.21249/) (2026) - [Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving](https://mlsanity.com/paper/2609.01659/) (2026) ## Abstract (arXiv) Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assumption is that successive VLM generations will continually improve driving performance and eventually outperform state-of-the-art methods. To systematically examine this assumption, we introduce LightEMMA, a longitudinal framework for evaluating the autonomous driving performance of VLMs. LightEMMA uses a lightweight, unified evaluation protocol that assesses each model's intrinsic driving capability without model-specific fine-tuning, architectural changes, or prompt engineering. Using this protocol, we evaluate 15 models from five major families on the challenging nuScenes prediction benchmark. Empirical findings show that, despite increased model scale and enhanced general reasoning capabilities, successive VLM generations do not consistently achieve better driving performance. Further analysis of driving scenarios reveals recurring failure modes, including overreliance on historical actions and difficulty reconciling conflicting visual cues. These findings highlight the need for domain-specific adaptation to improve the safety of VLM-based autonomous driving systems. The source code is available at https://github.com/michigan-traffic-lab/LightEMMA.