mlsanityData to 7 Oct 2026

LeGo-Drive: Language-enhanced Goal-oriented Closed-Loop End-to-End Autonomous Driving

arXiv 2403.20116 ↗First posted 29 Mar 2024cs.ROcited by 12

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Existing Vision-Language models (VLMs) estimate either long-term trajectory waypoints or a set of control actions as a reactive solution for closed-loop planning based on their rich scene comprehension. However, these estimations are coarse and are subjective to their "world understanding" which may generate sub-optimal decisions due to perception errors. In this paper, we introduce LeGo-Drive, which aims to address this issue by estimating a goal location based on the given language command as an intermediate representation in an end-to-end setting. The estimated goal might fall in a non-desirable region, like on top of a car for a parking-like command, leading to inadequate planning. Hence, we propose to train the architecture in an end-to-end manner, resulting in iterative refinement of both the goal and the trajectory collectively. We validate the effectiveness of our method through comprehensive experiments conducted in diverse simulated environments. We report significant improvements in standard autonomous driving metrics, with a goal reaching Success Rate of 81%. We further showcase the versatility of LeGo-Drive across different driving scenarios and linguistic inputs, underscoring its potential for practical deployment in autonomous vehicles and intelligent transportation systems.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 30 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

26 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2022Planning-oriented Autonomous Drivingcited by 1,068
  7. 2019Differentiable Convex Optimization Layerscited by 458
  8. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  9. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  10. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  11. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  12. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
12 more
  1. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  2. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  3. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  4. 2021DC3: A learning method for optimization with hard constraintscited by 181
  5. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  6. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  7. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  8. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  9. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  10. 2022Theseus: A Library for Differentiable Nonlinear Optimizationcited by 64
  11. 2021Grounding Linguistic Commands to Navigable Regionscited by 6
  12. 2023End-to-End Learning of Behavioural Inputs for Autonomous Driving in Dense Trafficcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 4.

12 papers.

  1. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  2. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  3. 2025PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Drivingcited by 16
  4. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agentscited by 15
  5. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  6. 2025ObjectReact: Learning Object-Relative Control for Visual Navigationcited by 5
  7. 2026ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Drivingcited by 3
  8. 2025Diffusion-FS: Multimodal Free-Space Prediction via Diffusion for Autonomous Drivingcited by 2
  9. 2026CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Drivingcited by 2
  10. 2025DIV-Nav: Open-Vocabulary Spatial Relationships for Multi-Object Navigationcited by 1
  11. 2026DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidancecited by 0
  12. 2026Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  2. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation
  3. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
  4. 2025From Human Intention to Action Prediction: Intention-Driven End-to-End Autonomous Driving
  5. 2026Vega: Learning to Drive with Natural Language Instructions
  6. 2025DriveMind: A Dual Visual Language Model-based Reinforcement Learning Framework for Autonomous Driving
  7. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
  8. 2026Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms