mlsanityData to 7 Oct 2026

VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenes

arXiv 2405.05885 ↗First posted 9 May 2024cs.ROcited by 22

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

Recent research on Large Language Models for autonomous driving shows promise in planning and control. However, high computational demands and hallucinations still challenge accurate trajectory prediction and control signal generation. Deterministic algorithms offer reliability but lack adaptability to complex driving scenarios and struggle with context and uncertainty. To address this problem, we propose VLM-Auto, a novel autonomous driving assistant system to empower the autonomous vehicles with adjustable driving behaviors based on the understanding of road scenes. A pipeline involving the CARLA simulator and Robot Operating System 2 (ROS2) verifying the effectiveness of our system is presented, utilizing a single Nvidia 4090 24G GPU while exploiting the capacity of textual output of the Visual Language Model (VLM). Besides, we also contribute a dataset containing an image set and a corresponding prompt set for fine-tuning the VLM module of our system. In CARLA experiments, our system achieved 97.82% average precision on 5 types of labels in our dataset. In the real-world driving dataset, our system achieved 96.97% prediction accuracy in night scenes and gloomy scenes. Our VLM-Auto dataset will be released at https://github.com/ZionGo6/VLM-Auto.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 22 of its 26 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

22 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024HawkDrive: A Transformer-driven Visual Perception System for Autonomous Driving in Night Scene6 Apr
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  3. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving8 Jan
  4. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  5. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoning17 May

Other references · 17

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  5. 2022Planning-oriented Autonomous Drivingcited by 1,068
  6. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  7. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  8. 2022Robot Operating System 2: Design, Architecture, and Uses In The Wildcited by 190
  9. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  10. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  11. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  12. 2022COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehiclescited by 95
5 more
  1. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  2. 2023LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoningcited by 59
  3. 2024Driving Everywhere with Large Language Model Policy Adaptationcited by 53
  4. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  5. 2024CognitiveDog: Large Multimodal Model Based System to Translate Vision and Language into Action of Quadruped Robotcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 7.

22 papers.

Sort:

By area:

  1. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  2. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  3. 2025ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Modelscited by 8
  4. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  5. 2025CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refinecited by 7
  6. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planningcited by 5
  7. 2024Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarkingcited by 3
  8. 2025Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactionscited by 3
  9. 2025Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysiscited by 3
  10. 2024METDrive: Multi-modal End-to-end Autonomous Driving with Temporal Guidancecited by 2
  11. 2025SimpleVSF: VLM-Scoring Fusion for Trajectory Prediction of End-to-End Autonomous Drivingcited by 2
  12. 2026Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theorycited by 2
10 more
  1. 2025All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehiclescited by 1
  2. 2026Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Modelscited by 1
  3. 2024OWLed: Outlier-weighed Layerwise Pruning for Efficient Autonomous Driving Frameworkcited by 0
  4. 2024A Novel MLLM-based Approach for Autonomous Driving in Different Weather Conditionscited by 0
  5. 2025A Low-Rank Method for Vision Language Model Hallucination Mitigation in Autonomous Drivingcited by 0
  6. 2026Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Drivingcited by 0
  7. 2026Feeling the Space: Egomotion-Aware Video Representation for Efficient and Accurate 3D Scene Understandingcited by 0
  8. 2026EagleVision: A Multi-Task Benchmark for Cross-Domain Perception in High-Speed Autonomous Racingcited by 0
  9. 2026Adversarial Attacks Against MLLMs via Progressive Resolution Processing and Adaptive Feature Alignmentcited by 0
  10. 2026Mamba-based Selective State Space Modeling Improves the Accuracy-Complexity Tradeoff of SmolVLA Vision-Language-Action Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process
  2. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision
  3. 2025A Hierarchical Test Platform for Vision Language Model (VLM)-Integrated Real-World Autonomous Driving
  4. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  5. 2025dVLM-AD: Enhance Diffusion Vision-Language-Model for Driving via Controllable Reasoning
  6. 2024On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation
  7. 2025VLMPlanner: Integrating Visual Language Models with Motion Planning
  8. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving