mlsanityData to 14 Oct 2026

2nd Place Solution for CVPR2024 E2E Challenge: End-to-End Autonomous Driving Using Vision Language Model

arXiv 2509.02659 ↗First posted 2 Sep 2025cs.CV cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

End-to-end autonomous driving has drawn tremendous attention recently. Many works focus on using modular deep neural networks to construct the end-to-end archi-tecture. However, whether using powerful large language models (LLM), especially multi-modality Vision Language Models (VLM) could benefit the end-to-end driving tasks remain a question. In our work, we demonstrate that combining end-to-end architectural design and knowledgeable VLMs yield impressive performance on the driving tasks. It is worth noting that our method only uses a single camera and is the best camera-only solution across the leaderboard, demonstrating the effectiveness of vision-based driving approach and the potential for end-to-end driving tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 25 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

23 of 25 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  2. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models2 Jan
  3. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  4. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 18

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Planning-oriented Autonomous Drivingcited by 1,087
  4. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  5. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  6. 2024InternLM2 Technical Reportcited by 397
  7. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 368
  8. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  9. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  10. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  11. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  12. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
6 more
  1. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 191
  2. 2023A Language Agent for Autonomous Drivingcited by 164
  3. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  4. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  5. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion modelcited by 75
  6. 2024GraphAD: Interaction Scene Graph for End-to-end Autonomous Drivingcited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  2. 2026AppleVLM: End-to-end Autonomous Driving with Advanced Perception and Planning-Enhanced Vision-Language Models
  3. 2025SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving
  4. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision
  5. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
  6. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
  7. 2024SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving
  8. 2026A Collaborative Multi-Modality Interaction for VLA-based End-to-End Autonomous Driving