mlsanityData to 7 Oct 2026

BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Driving

arXiv 2503.03074 ↗First posted 5 Mar 2025cs.RO cs.CVcited by 26

On the map: End-to-end autonomous driving → VLMs for autonomous driving

Abstract (arXiv)

Autonomous driving has the potential to set the stage for more efficient future mobility, requiring the research domain to establish trust through safe, reliable and transparent driving. Large Language Models (LLMs) possess reasoning capabilities and natural language understanding, presenting the potential to serve as generalized decision-makers for ego-motion planning that can interact with humans and navigate environments designed for human drivers. While this research avenue is promising, current autonomous driving approaches are challenged by combining 3D spatial grounding and the reasoning and language capabilities of LLMs. We introduce BEVDriver, an LLM-based model for end-to-end closed-loop driving in CARLA that utilizes latent BEV features as perception input. BEVDriver includes a BEV encoder to efficiently process multi-view images and 3D LiDAR point clouds. Within a common latent space, the BEV features are propagated through a Q-Former to align with natural language instructions and passed to the LLM that predicts and plans precise future trajectories while considering navigation instructions and critical scenarios. On the LangAuto benchmark, our model reaches up to 18.9% higher performance on the Driving Score compared to SoTA methods.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving. Builds on 30 of its 34 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

30 of 34 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024CarLLaVA: Vision language models for camera-only closed-loop driving14 Jun
  2. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  3. 2024Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models2 Jan
  4. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  5. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving3 Oct
  6. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoning17 May

Other references · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2016Pyramid Scene Parsing Networkcited by 2,472
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2022Planning-oriented Autonomous Drivingcited by 1,068
  7. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  8. 2022BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detectioncited by 568
  9. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  10. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  11. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  12. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
12 more
  1. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  2. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  3. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  4. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  5. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  6. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  7. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  8. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  9. 2024DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mambacited by 87
  10. 2020BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloudcited by 38
  11. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  12. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 16.

26 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  3. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  4. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  5. 2026Unifying Language-Action Understanding and Generation for Autonomous Drivingcited by 9
  6. 2025GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Drivingcited by 6
  7. 2026HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Drivingcited by 5
  8. 2026MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systemscited by 4
  9. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  10. 2025Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognitioncited by 3
  11. 2026ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Drivingcited by 3
  12. 2025SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Drivingcited by 2
14 more
  1. 2026LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformerscited by 2
  2. 2026VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Eventscited by 2
  3. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  4. 2025HABIT: Human Action Benchmark for Interactive Traffic in CARLAcited by 1
  5. 2025Closing the Navigation Compliance Gap in End-to-end Autonomous Drivingcited by 1
  6. 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Drivingcited by 1
  7. 2026What's Hidden Matters: Identifying Planning-Critical Occluded Agents using Vision-Language Modelscited by 1
  8. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representationscited by 0
  9. 2026BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusioncited by 0
  10. 2026WorldVLM: Combining World Model Forecasting and Vision-Language Reasoningcited by 0
  11. 2026TPS-Drive: Task-Guided Representation Purification for VLM-based Autonomous Drivingcited by 0
  12. 2026TGRIP: A Text-Guided Approach to Vehicle Instance Prediction in Autonomous Drivingcited by 0
  13. 2026MoRAL: Sensor-Grounded BEV Reasoning for Compact VLMs toward Edge-Oriented Autonomous Drivingcited by 0
  14. 2026Towards Zero-Shot Transfer Across Embodiments For Driving VLAscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models
  2. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving
  3. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents
  4. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  5. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving
  6. 2026GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving
  7. 2025Goal-Based Vision-Language Driving
  8. 2025BEV-VLM: Trajectory Planning via Unified BEV Abstraction