mlsanityData to 14 Oct 2026

SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving

arXiv 2505.16805 ↗First posted 22 May 2025cs.CVcited by 21

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

The integration of Vision-Language Models (VLMs) into autonomous driving systems has shown promise in addressing key challenges such as learning complexity, interpretability, and common-sense reasoning. However, existing approaches often struggle with efficient integration and realtime decision-making due to computational demands. In this paper, we introduce SOLVE, an innovative framework that synergizes VLMs with end-to-end (E2E) models to enhance autonomous vehicle planning. Our approach emphasizes knowledge sharing at the feature level through a shared visual encoder, enabling comprehensive interaction between VLM and E2E components. We propose a Trajectory Chain-of-Thought (T-CoT) paradigm, which progressively refines trajectory predictions, reducing uncertainty and improving accuracy. By employing a temporal decoupling strategy, SOLVE achieves efficient cooperation by aligning high-quality VLM outputs with E2E real-time performance. Evaluated on the nuScenes dataset, our method demonstrates significant improvements in trajectory prediction accuracy, paving the way for more robust and reliable autonomous driving systems.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 51 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

41 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving29 Oct
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  3. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 37

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  10. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  11. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  12. 2022Planning-oriented Autonomous Drivingcited by 1,087
25 more
  1. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 622
  2. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 537
  3. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  4. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  5. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  6. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 315
  7. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 295
  8. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 289
  9. 2019Causal Confusion in Imitation Learningcited by 288
  10. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  11. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 241
  12. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  13. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  14. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 212
  15. 2021Multimodal Motion Prediction with Stacked Transformerscited by 208
  16. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  17. 2023A Language Agent for Autonomous Drivingcited by 164
  18. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 160
  19. 2020Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Reviewcited by 116
  20. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 114
  21. 2020PiP: Planning-informed Trajectory Prediction for Autonomous Drivingcited by 80
  22. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  23. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  24. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  25. 2024Exploring the Causality of End-to-End Autonomous Drivingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 9.

21 papers.

Sort:

By area:

  1. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  2. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  3. 2026DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Drivingcited by 19
  4. 2025PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Drivingcited by 16
  5. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Drivingcited by 11
  6. 2026SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse Worldcited by 8
  7. 2025CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refinecited by 7
  8. 2025GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Drivingcited by 7
  9. 2026SparseOccVLA: Bridging Occupancy and Vision-Language Models via Sparse Queries for Unified 4D Scene Understanding and Planningcited by 4
  10. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Surveycited by 3
  11. 2025ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Expertscited by 3
  12. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
9 more
  1. 2025MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusioncited by 3
  2. 2026OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Modelscited by 3
  3. 2026XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environmentscited by 2
  4. 2025KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Modelscited by 1
  5. 2025ClimaOoD: Improving Anomaly Segmentation via Physically Realistic Synthetic Datacited by 0
  6. 2026Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Drivingcited by 0
  7. 2026VeriDrive: Verifiable Counterfactual Supervision for Cost-Efficient Vision-Language Planningcited by 0
  8. 2026DrivingAgent: Design and Scheduling Agents for Autonomous Driving Systemscited by 0
  9. 2026FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation
  2. 2026VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
  3. 2025VLMPlanner: Integrating Visual Language Models with Motion Planning
  4. 2024V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models
  5. 2026NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Driving
  6. 2025CoT4AD: A Vision-Language-Action Model with Explicit Chain-of-Thought Reasoning for Autonomous Driving
  7. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
  8. 2024VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision