# SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving

arXiv: https://arxiv.org/abs/2505.16805 · HTML: https://mlsanity.com/paper/2505.16805/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 51 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)
- End-to-end autonomous driving / [Reasoning VLA and VLM driving policies](https://mlsanity.com/driving/reasoning-vla-and-vlm-driving-policies/)

## Builds on (most cited first, 30 of 41)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?](https://mlsanity.com/paper/2312.03031/) (2023)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving](https://mlsanity.com/paper/2310.01957/) (2023)
- [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)

## Built on by (most cited first, 21 of 21)

- [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025)
- [SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving](https://mlsanity.com/paper/2512.10719/) (2025)
- [DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Driving](https://mlsanity.com/paper/2602.14577/) (2026)
- [PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Driving](https://mlsanity.com/paper/2507.17596/) (2025)
- [ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving](https://mlsanity.com/paper/2512.22939/) (2025)
- [SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse World](https://mlsanity.com/paper/2602.18887/) (2026)
- [CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refine](https://mlsanity.com/paper/2509.15968/) (2025)
- [GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving](https://mlsanity.com/paper/2511.11266/) (2025)
- [SparseOccVLA: Bridging Occupancy and Vision-Language Models via Sparse Queries for Unified 4D Scene Understanding and Planning](https://mlsanity.com/paper/2601.06474/) (2026)
- [Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2509.10570/) (2025)
- [ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Experts](https://mlsanity.com/paper/2511.11740/) (2025)
- [Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges](https://mlsanity.com/paper/2512.00021/) (2025)
- [MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion](https://mlsanity.com/paper/2512.13177/) (2025)
- [OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models](https://mlsanity.com/paper/2604.17915/) (2026)
- [XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environments](https://mlsanity.com/paper/2604.18484/) (2026)
- [KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Models](https://mlsanity.com/paper/2509.02966/) (2025)
- [ClimaOoD: Improving Anomaly Segmentation via Physically Realistic Synthetic Data](https://mlsanity.com/paper/2512.02686/) (2025)
- [Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving](https://mlsanity.com/paper/2605.24004/) (2026)
- [VeriDrive: Verifiable Counterfactual Supervision for Cost-Efficient Vision-Language Planning](https://mlsanity.com/paper/2606.07338/) (2026)
- [DrivingAgent: Design and Scheduling Agents for Autonomous Driving Systems](https://mlsanity.com/paper/2606.12236/) (2026)
- [FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memory](https://mlsanity.com/paper/2609.18623/) (2026)

## Abstract (arXiv)

The integration of Vision-Language Models (VLMs) into autonomous driving systems has shown promise in addressing key challenges such as learning complexity, interpretability, and common-sense reasoning. However, existing approaches often struggle with efficient integration and realtime decision-making due to computational demands. In this paper, we introduce SOLVE, an innovative framework that synergizes VLMs with end-to-end (E2E) models to enhance autonomous vehicle planning. Our approach emphasizes knowledge sharing at the feature level through a shared visual encoder, enabling comprehensive interaction between VLM and E2E components. We propose a Trajectory Chain-of-Thought (T-CoT) paradigm, which progressively refines trajectory predictions, reducing uncertainty and improving accuracy. By employing a temporal decoupling strategy, SOLVE achieves efficient cooperation by aligning high-quality VLM outputs with E2E real-time performance. Evaluated on the nuScenes dataset, our method demonstrates significant improvements in trajectory prediction accuracy, paving the way for more robust and reliable autonomous driving systems.