# A Comprehensive LLM-powered Framework for Driving Intelligence Evaluation

arXiv: https://arxiv.org/abs/2503.05164 · HTML: https://mlsanity.com/paper/2503.05164/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 30 of 34)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023)
- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving](https://mlsanity.com/paper/2310.01957/) (2023)
- [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023)
- [Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge](https://mlsanity.com/paper/2407.19594/) (2024)
- [MotionLM: Multi-Agent Motion Forecasting as Language Modeling](https://mlsanity.com/paper/2309.16534/) (2023)
- [LingoQA: Visual Question Answering for Autonomous Driving](https://mlsanity.com/paper/2312.14115/) (2023)
- [VLP: Vision Language Planning for Autonomous Driving](https://mlsanity.com/paper/2401.05577/) (2024)
- [RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model](https://mlsanity.com/paper/2402.10828/) (2024)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases](https://mlsanity.com/paper/2404.10595/) (2024)
- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles](https://mlsanity.com/paper/2206.09682/) (2022)
- [LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoning](https://mlsanity.com/paper/2401.00125/) (2023)
- [Personalized Autonomous Driving with Large Language Models: Field Experiments](https://mlsanity.com/paper/2312.09397/) (2023)
- [DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving](https://mlsanity.com/paper/2401.03641/) (2024)
- [Human-Centered Design Recommendations for LLM-as-a-Judge](https://mlsanity.com/paper/2407.03479/) (2024)
- [LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Driving](https://mlsanity.com/paper/2402.01246/) (2024)
- [DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiences](https://mlsanity.com/paper/2406.03008/) (2024)
- [Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?](https://mlsanity.com/paper/2106.12233/) (2021)
- [Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF](https://mlsanity.com/paper/2406.04481/) (2024)

## Built on by (most cited first, 5 of 5)

- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models](https://mlsanity.com/paper/2510.13108/) (2025)
- [RISEE: A Highly Interactive Naturalistic Driving Trajectories Dataset with Human Subjective Risk Perception and Eye-tracking Information](https://mlsanity.com/paper/2507.19490/) (2025)

## Abstract (arXiv)

Evaluation methods for autonomous driving are crucial for algorithm optimization. However, due to the complexity of driving intelligence, there is currently no comprehensive evaluation method for the level of autonomous driving intelligence. In this paper, we propose an evaluation framework for driving behavior intelligence in complex traffic environments, aiming to fill this gap. We constructed a natural language evaluation dataset of human professional drivers and passengers through naturalistic driving experiments and post-driving behavior evaluation interviews. Based on this dataset, we developed an LLM-powered driving evaluation framework. The effectiveness of this framework was validated through simulated experiments in the CARLA urban traffic simulator and further corroborated by human assessment. Our research provides valuable insights for evaluating and designing more intelligent, human-like autonomous driving agents. The implementation details of the framework and detailed information about the dataset can be found at Github.