mlsanityData to 14 Oct 2026

A Comprehensive LLM-powered Framework for Driving Intelligence Evaluation

arXiv 2503.05164 ↗First posted 7 Mar 2025cs.RO cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Evaluation methods for autonomous driving are crucial for algorithm optimization. However, due to the complexity of driving intelligence, there is currently no comprehensive evaluation method for the level of autonomous driving intelligence. In this paper, we propose an evaluation framework for driving behavior intelligence in complex traffic environments, aiming to fill this gap. We constructed a natural language evaluation dataset of human professional drivers and passengers through naturalistic driving experiments and post-driving behavior evaluation interviews. Based on this dataset, we developed an LLM-powered driving evaluation framework. The effectiveness of this framework was validated through simulated experiments in the CARLA urban traffic simulator and further corroborated by human assessment. Our research provides valuable insights for evaluating and designing more intelligent, human-like autonomous driving agents. The implementation details of the framework and detailed information about the dataset can be found at Github.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

34 of 49 references.

References · 34

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  4. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  5. 2024Self-Rewarding Language Modelscited by 681
  6. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  7. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  8. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  9. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  10. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  11. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  12. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
22 more
  1. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 207
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  3. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 156
  4. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 145
  5. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 126
  6. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 92
  7. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  8. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 77
  9. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  10. 2022SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehiclescited by 66
  11. 2023LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoningcited by 59
  12. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 48
  13. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 43
  14. 2024Human-Centered Design Recommendations for LLM-as-a-Judgecited by 37
  15. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 33
  16. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  17. 2021Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?cited by 19
  18. 2024Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHFcited by 18
  19. 2024Driving Style Alignment for LLM-powered Driver Agentcited by 15
  20. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14
  21. 2018Is it Safe to Drive? An Overview of Factors, Challenges, and Datasets for Driveability Assessment in Autonomous Drivingcited by 11
  22. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehiclescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  2. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  3. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  4. 2025DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Modelscited by 3
  5. 2025RISEE: A Highly Interactive Naturalistic Driving Trajectories Dataset with Human Subjective Risk Perception and Eye-tracking Informationcited by 1