mlsanityData to 7 Oct 2026

SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment

arXiv 2503.09594 ↗First posted 12 Mar 2025cs.CV cs.ROcited by 149

On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies

Abstract (arXiv)

Integrating large language models (LLMs) into autonomous driving has attracted significant attention with the hope of improving generalization and explainability. However, existing methods often focus on either driving or vision-language understanding but achieving both high driving performance and extensive language understanding remains challenging. In addition, the dominant approach to tackle vision-language understanding is using visual question answering. However, for autonomous driving, this is only useful if it is aligned with the action space. Otherwise, the model's answers could be inconsistent with its behavior. Therefore, we propose a model that can handle three different tasks: (1) closed-loop driving, (2) vision-language understanding, and (3) language-action alignment. Our model SimLingo is based on a vision language model (VLM) and works using only camera, excluding expensive sensors like LiDAR. SimLingo obtains state-of-the-art performance on the widely used CARLA simulator on the Bench2Drive benchmark and is the winning entry at the CARLA challenge 2024. Additionally, we achieve strong results in a wide variety of language-related tasks while maintaining high driving performance.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 52 of its 68 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

52 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024EMMA: End-to-End Multimodal Model for Autonomous Driving30 Oct
  2. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving25 Mar
  3. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)26 Feb

Other references · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  11. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  12. 2024Qwen2 Technical Reportcited by 2,076
37 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  3. 2022Planning-oriented Autonomous Drivingcited by 1,068
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  6. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  7. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  8. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  9. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  10. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  11. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  12. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  13. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  14. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  15. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  16. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  17. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  18. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  19. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  20. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  21. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  22. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  23. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  24. 2023A Language Agent for Autonomous Drivingcited by 161
  25. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  26. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  27. 2022Learning from All Vehiclescited by 135
  28. 2021Learning to drive from a world on railscited by 114
  29. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  30. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  31. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  32. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  33. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  34. 2024Hidden Biases of End-to-End Driving Datasetscited by 30
  35. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Drivingcited by 22
  36. 2024Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Drivingcited by 11
  37. 2025A Comprehensive Evaluation of Four End-to-End AI Autopilots Using CCTest and the Carla Leaderboardcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 35; 2026: 114.

149 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  3. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  4. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  5. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  6. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  7. 2025Generalized Trajectory Scoring for End-to-end Multimodal Planningcited by 63
  8. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  9. 2025Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Trainingcited by 39
  10. 2026SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Drivingcited by 34
  11. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  12. 2025IRL-VLA: Training an Vision-Language-Action Policy via Reward World Modelcited by 31
38 more
  1. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  2. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  3. 2025MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learningcited by 25
  4. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
  5. 2025Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Drivingcited by 24
  6. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  7. 2025LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Drivingcited by 22
  8. 2025Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoningcited by 21
  9. 2026SparseDriveV2: Scoring is All You Need for End-to-End Autonomous Drivingcited by 21
  10. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  11. 2025BridgeDrive: Diffusion Bridge Policy for Closed-Loop Trajectory Planning in Autonomous Drivingcited by 17
  12. 2026UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Drivingcited by 17
  13. 2026Fail2Drive: Benchmarking Closed-Loop Driving Generalizationcited by 15
  14. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 12
  15. 2026DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Drivingcited by 12
  16. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  17. 2026DynVLA: Learning World Dynamics for Action Reasoning in Autonomous Drivingcited by 11
  18. 2026MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Drivingcited by 10
  19. 2026Unifying Language-Action Understanding and Generation for Autonomous Drivingcited by 9
  20. 2026AutoMoT: A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Drivingcited by 9
  21. 2025PlanT 2.0: Exposing Biases and Structural Flaws in Closed-Loop Drivingcited by 8
  22. 2026Learning from Mistakes: Post-Training for Driving VLA with Takeover Datacited by 8
  23. 2026Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Modelscited by 8
  24. 2026nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Drivingcited by 8
  25. 2026SafeDrive: Fine-Grained Safety Reasoning for End-to-End Driving in a Sparse Worldcited by 7
  26. 2026Test-Time Trajectory Optimization for Autonomous Drivingcited by 7
  27. 2025ALN-P3: Unified Language Alignment for Perception, Prediction, and Planning in Autonomous Drivingcited by 6
  28. 2025GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Drivingcited by 6
  29. 2026Generative Scenario Rollouts for End-to-End Autonomous Drivingcited by 6
  30. 2026Learning to Drive in New Cities Without Human Demonstrationscited by 6
  31. 2026Vega: Learning to Drive with Natural Language Instructionscited by 6
  32. 2026Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drivecited by 6
  33. 2026MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Drivingcited by 6
  34. 2026Natural Language Instructions for Scene-Responsive Human-in-the-Loop Motion Planning in Autonomous Driving using Vision-Language-Action Modelscited by 5
  35. 2026HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Drivingcited by 5
  36. 2026StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Drivingcited by 5
  37. 2026What Matters for Scalable and Robust Learning in End-to-End Driving Planners?cited by 5
  38. 2026Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Drivingcited by 5

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  2. 2024CarLLaVA: Vision language models for camera-only closed-loop driving
  3. 2026Rethinking Language's Role in Efficient VLA for Autonomous Vehicles: Toward Smarter, Trustworthy Driving
  4. 2023LingoQA: Visual Question Answering for Autonomous Driving
  5. 2025LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving
  6. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  7. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model
  8. 20252nd Place Solution for CVPR2024 E2E Challenge: End-to-End Autonomous Driving Using Vision Language Model