mlsanityData to 7 Oct 2026

V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models

arXiv 2408.09251 ↗First posted 17 Aug 2024cs.RO cs.AI cs.LGcited by 47

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Vehicle-to-everything (V2X) cooperation has emerged as a promising paradigm to overcome the perception limitations of classical autonomous driving by leveraging information from both ego-vehicle and infrastructure sensors. However, effectively fusing heterogeneous visual and semantic information while ensuring robust trajectory planning remains a significant challenge. This paper introduces V2X-VLM, a novel end-to-end (E2E) cooperative autonomous driving framework based on vision-language models (VLMs). V2X-VLM integrates multiperspective camera views from vehicles and infrastructure with text-based scene descriptions to enable a more comprehensive understanding of driving environments. Specifically, we propose a contrastive learning-based mechanism to reinforce the alignment of heterogeneous visual and textual characteristics, which enhances the semantic understanding of complex driving scenarios, and employ a knowledge distillation strategy to stabilize training. Experiments on a large real-world dataset demonstrate that V2X-VLM achieves state-of-the-art trajectory planning accuracy, significantly reducing L2 error and collision rate compared to existing cooperative autonomous driving baselines. Ablation studies validate the contributions of each component. Moreover, the evaluation of robustness and efficiency highlights the practicality of V2X-VLM for real-world deployment to enhance overall autonomous driving safety and decision-making.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 76 references that are in the map. First posted on arXiv 17 Aug 2024.

Builds on

52 of 76 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?28 May
  2. 2024Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving28 Mar
  3. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  4. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  5. 2023Dolphins: Multimodal Language Model for Driving1 Dec
  6. 2023A Survey on Multimodal Large Language Models for Autonomous Driving21 Nov
  7. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 45

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2022Planning-oriented Autonomous Drivingcited by 1,068
  4. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  5. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  6. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  7. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  8. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  9. 2022V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformercited by 320
  10. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  11. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  12. 2020V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Predictioncited by 292
33 more
  1. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  2. 2022DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detectioncited by 270
  3. 2022Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Mapscited by 255
  4. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  5. 2024Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillationcited by 244
  6. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  7. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  8. 2022CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformerscited by 227
  9. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  10. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  11. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  12. 2022Robust Collaborative 3D Object Detection in Presence of Pose Errorscited by 129
  13. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  14. 2023V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecastingcited by 114
  15. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 112
  16. 2022COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehiclescited by 95
  17. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  18. 2024DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mambacited by 87
  19. 2023Collaboration Helps Camera Overtake LiDAR in 3D Detectioncited by 81
  20. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  21. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  22. 2022Occupancy Flow Fields for Motion Forecasting in Autonomous Drivingcited by 62
  23. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  24. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  25. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Modelcited by 44
  26. 2024End-to-End Autonomous Driving through V2X Cooperationcited by 42
  27. 2022Multi-stage image denoising with the wavelet transformcited by 37
  28. 2024HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Drivingcited by 31
  29. 2024End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotationcited by 27
  30. 2023Interruption-Aware Cooperative Perception for V2X Communication-Aided Autonomous Drivingcited by 25
  31. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  32. 2023Exploring Multi-Timestep Multi-Stage Diffusion Features for Hyperspectral Image Classificationcited by 15
  33. 2023Contrastive Vision-Language Alignment Makes Efficient Instruction Learnercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 30; 2026: 14.

47 papers.

Sort:

By area:

  1. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  2. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  3. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  4. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  5. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  6. 2025LangCoop: Collaborative Driving with Languagecited by 25
  7. 2025A Survey on Efficient Vision-Language Modelscited by 19
  8. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  9. 2025O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Modelscited by 14
  10. 2025V2X-UniPool: Unifying Multimodal Perception and Knowledge Reasoning for Autonomous Drivingcited by 14
  11. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  12. 2025Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Modelscited by 13
35 more
  1. 2025V2X-LLM: Enhancing V2X Integration and Understanding in Connected Vehicle Corridorscited by 13
  2. 2025Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulationcited by 13
  3. 2025AirV2X: Unified Air-Ground Vehicle-to-Everything Collaborationcited by 11
  4. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 11
  5. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Groundingcited by 10
  6. 2025Research Challenges and Progress in the End-to-End V2X Cooperative Autonomous Driving Competitioncited by 9
  7. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  8. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  9. 2025VLM-Assisted Continual learning for Visual Question Answering in Self-Drivingcited by 6
  10. 2025DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Drivingcited by 5
  11. 2025Automated Vehicles Should be Connected with Natural Languagecited by 5
  12. 2025Risk Map As Middleware: Towards Interpretable Cooperative End-to-end Autonomous Driving for Risk-Aware Planningcited by 5
  13. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planningcited by 5
  14. 2026HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Drivingcited by 5
  15. 2025Spatial-aware Vision Language Model for Autonomous Drivingcited by 4
  16. 2025Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidancecited by 3
  17. 2025A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Modelscited by 3
  18. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  19. 2025SimpleVSF: VLM-Scoring Fusion for Trajectory Prediction of End-to-End Autonomous Drivingcited by 2
  20. 2025SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Drivingcited by 2
  21. 2026Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoningcited by 2
  22. 2026OmniV2X: A Generative Foundation Planner for Efficient End-to-End Cooperative Drivingcited by 2
  23. 2026OpenCOOD-Air: Prompting Heterogeneous Ground-Air Collaborative Perception with Spatial Conversion and Offset Predictioncited by 1
  24. 2026DH-VLM: Dual-Horizon Cooperative Latent Reasoning for Autonomous Drivingcited by 1
  25. 2025Sky-Drive: A Distributed Multi-Agent Simulation Platform for Human-AI Collaborative and Socially-Aware Future Transportationcited by 0
  26. 2025A Concept for Efficient Scalability of Automated Driving Allowing for Technical, Legal, Cultural, and Ethical Differencescited by 0
  27. 2026Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Drivingcited by 0
  28. 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Makingcited by 0
  29. 2026TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarioscited by 0
  30. 2026LACO: Adaptive Latent Communication for Collaborative Drivingcited by 0
  31. 2026Large Language Models in Transportation Systems Management and Operations: From Text Reasoning to Multi-modal Decision Supportcited by 0
  32. 2026TALSC: Timeliness-Aware Large-Small VLM Collaboration for Infrastructure-Assisted Autonomous Drivingcited by 0
  33. 2026G-MARK: Grounded Multi-Agent Reasoning for Cooperative Driving via Knowledge Graphscited by 0
  34. 2026InfraVLA: Extending Vision-Language-Action Navigation with Infrastructure Camerascited by 0
  35. 2026CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Roadside-Cooperative Autonomous Driving: From Data Platform to Vision-Language End-to-End Reasoning
  2. 2025V2V-GoT: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models and Graph-of-Thoughts
  3. 2025SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving
  4. 2025V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models
  5. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process
  6. 2024Unified End-to-End V2X Cooperative Autonomous Driving
  7. 2026CMU-Drive and V2V-VLA: Cooperative Multi-agent Unified Driving with Reasoning Benchmark and Vehicle-to-Vehicle Vision-Language-Action Models
  8. 2025VLMPlanner: Integrating Visual Language Models with Motion Planning