# BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Driving

arXiv: https://arxiv.org/abs/2503.03074 · HTML: https://mlsanity.com/paper/2503.03074/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 30 of its 34 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Placed in

- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 30 of 30)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [GenAD: Generalized Predictive Model for Autonomous Driving](https://mlsanity.com/paper/2403.09630/) (2024)
- [Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe](https://mlsanity.com/paper/2209.05324/) (2022)
- [ReasonNet: End-to-End Driving with Temporal and Global Reasoning](https://mlsanity.com/paper/2305.10507/) (2023)
- [RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model](https://mlsanity.com/paper/2402.10828/) (2024)
- [Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving](https://mlsanity.com/paper/2310.02251/) (2023)
- [DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mamba](https://mlsanity.com/paper/2408.03601/) (2024)
- [Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models](https://mlsanity.com/paper/2401.00988/) (2024)
- [CarLLaVA: Vision language models for camera-only closed-loop driving](https://mlsanity.com/paper/2406.10165/) (2024)
- [BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud](https://mlsanity.com/paper/2006.11436/) (2020)
- [Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?](https://mlsanity.com/paper/2404.07569/) (2024)
- [AD-H: Language-guided Autonomous Driving with Hierarchical Agents](https://mlsanity.com/paper/2406.03474/) (2024)

## Built on by (most cited first, 26 of 26)

- [AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.13757/) (2025)
- [A Survey on Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2506.24044/) (2025)
- [SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving](https://mlsanity.com/paper/2601.05640/) (2026)
- [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025)
- [Unifying Language-Action Understanding and Generation for Autonomous Driving](https://mlsanity.com/paper/2603.01441/) (2026)
- [GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving](https://mlsanity.com/paper/2511.11266/) (2025)
- [HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2602.13329/) (2026)
- [MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systems](https://mlsanity.com/paper/2605.10904/) (2026)
- [Generative AI for Autonomous Driving: A Review](https://mlsanity.com/paper/2505.15863/) (2025)
- [Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognition](https://mlsanity.com/paper/2511.14391/) (2025)
- [ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Driving](https://mlsanity.com/paper/2604.05378/) (2026)
- [SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Driving](https://mlsanity.com/paper/2510.18123/) (2025)
- [LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers](https://mlsanity.com/paper/2603.02035/) (2026)
- [VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events](https://mlsanity.com/paper/2603.18178/) (2026)
- [LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving](https://mlsanity.com/paper/2604.08719/) (2026)
- [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025)
- [HABIT: Human Action Benchmark for Interactive Traffic in CARLA](https://mlsanity.com/paper/2511.19109/) (2025)
- [Closing the Navigation Compliance Gap in End-to-end Autonomous Driving](https://mlsanity.com/paper/2512.10660/) (2025)
- [What's Hidden Matters: Identifying Planning-Critical Occluded Agents using Vision-Language Models](https://mlsanity.com/paper/2607.00283/) (2026)
- [BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representations](https://mlsanity.com/paper/2603.06576/) (2026)
- [BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusion](https://mlsanity.com/paper/2603.09961/) (2026)
- [WorldVLM: Combining World Model Forecasting and Vision-Language Reasoning](https://mlsanity.com/paper/2603.14497/) (2026)
- [TPS-Drive: Task-Guided Representation Purification for VLM-based Autonomous Driving](https://mlsanity.com/paper/2605.27038/) (2026)
- [TGRIP: A Text-Guided Approach to Vehicle Instance Prediction in Autonomous Driving](https://mlsanity.com/paper/2607.04812/) (2026)
- [MoRAL: Sensor-Grounded BEV Reasoning for Compact VLMs toward Edge-Oriented Autonomous Driving](https://mlsanity.com/paper/2608.02449/) (2026)
- [Towards Zero-Shot Transfer Across Embodiments For Driving VLAs](https://mlsanity.com/paper/2609.02341/) (2026)

## Abstract (arXiv)

Autonomous driving has the potential to set the stage for more efficient future mobility, requiring the research domain to establish trust through safe, reliable and transparent driving. Large Language Models (LLMs) possess reasoning capabilities and natural language understanding, presenting the potential to serve as generalized decision-makers for ego-motion planning that can interact with humans and navigate environments designed for human drivers. While this research avenue is promising, current autonomous driving approaches are challenged by combining 3D spatial grounding and the reasoning and language capabilities of LLMs. We introduce BEVDriver, an LLM-based model for end-to-end closed-loop driving in CARLA that utilizes latent BEV features as perception input. BEVDriver includes a BEV encoder to efficiently process multi-view images and 3D LiDAR point clouds. Within a common latent space, the BEV features are propagated through a Q-Former to align with natural language instructions and passed to the LLM that predicts and plans precise future trajectories while considering navigation instructions and critical scenarios. On the LangAuto benchmark, our model reaches up to 18.9% higher performance on the Driving Score compared to SoTA methods.