mlsanityData to 7 Oct 2026

Vision-Integrated LLMs for Autonomous Driving Assistance : Human Performance Comparison and Trust Evaluation

arXiv 2502.06843 ↗First posted 6 Feb 2025cs.CV cs.AI cs.HCcited by 0

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Traditional autonomous driving systems often struggle with reasoning in complex, unexpected scenarios due to limited comprehension of spatial relationships. In response, this study introduces a Large Language Model (LLM)-based Autonomous Driving (AD) assistance system that integrates a vision adapter and an LLM reasoning module to enhance visual understanding and decision-making. The vision adapter, combining YOLOv4 and Vision Transformer (ViT), extracts comprehensive visual features, while GPT-4 enables human-like spatial reasoning and response generation. Experimental evaluations with 45 experienced drivers revealed that the system closely mirrors human performance in describing situations and moderately aligns with human decisions in generating appropriate responses.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 33 of its 58 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

33 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model16 Feb
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  3. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Models12 Dec
  4. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  5. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  6. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  7. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2018YOLOv3: An Incremental Improvementcited by 2,652
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  11. 2016End to End Learning for Self-Driving Carscited by 1,000
  12. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
14 more
  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  2. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  3. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  4. 2018Learning to Drive in a Daycited by 266
  5. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  6. 2018Deep Learning: A Critical Appraisalcited by 214
  7. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  8. 2020Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehiclescited by 101
  9. 2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 99
  10. 2021YOLOP: You Only Look Once for Panoptic Driving Perceptioncited by 82
  11. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  12. 2019Human-Like Autonomous Car-Following Model with Deep Reinforcement Learningcited by 47
  13. 2021DQ-GAT: Towards Safe and Efficient Autonomous Driving with Deep Q-Learning and Graph Attention Networkscited by 13
  14. 2020Machine Ethics and Automated Vehiclescited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  2. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  3. 2024Leveraging Large Language Models for Enhancing Autonomous Vehicle Perception
  4. 2023Evaluation of Large Language Models for Decision Making in Autonomous Driving
  5. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  6. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
  7. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  8. 2026Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models