mlsanityData to 7 Oct 2026

CarLLaVA: Vision language models for camera-only closed-loop driving

arXiv 2406.10165 ↗First posted 14 Jun 2024cs.CV cs.ROcited by 57

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

In this technical report, we present CarLLaVA, a Vision Language Model (VLM) for autonomous driving, developed for the CARLA Autonomous Driving Challenge 2.0. CarLLaVA uses the vision encoder of the LLaVA VLM and the LLaMA architecture as backbone, achieving state-of-the-art closed-loop driving performance with only camera input and without the need for complex or expensive labels. Additionally, we show preliminary results on predicting language commentary alongside the driving output. CarLLaVA uses a semi-disentangled output representation of both path predictions and waypoints, getting the advantages of the path for better lateral control and the waypoints for better longitudinal control. We propose an efficient training recipe to train on large driving datasets without wasting compute on easy, trivial data. CarLLaVA ranks 1st place in the sensor track of the CARLA Autonomous Driving Challenge 2.0 outperforming the previous state of the art by 458% and the best concurrent submission by 32.6%.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 33 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

27 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  3. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  9. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  10. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  11. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  12. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
12 more
  1. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  2. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  3. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  4. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  5. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  6. 2022Learning from All Vehiclescited by 135
  7. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  8. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  9. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  10. 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
  11. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  12. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)cited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 32; 2026: 18.

57 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  3. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  4. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 82
  5. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 74
  6. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  7. 2025Generalized Trajectory Scoring for End-to-end Multimodal Planningcited by 63
  8. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  9. 2025Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Trainingcited by 39
  10. 2025HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decodercited by 35
  11. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34
  12. 2024Hidden Biases of End-to-End Driving Datasetscited by 30
38 more
  1. 2025BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Drivingcited by 26
  2. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  3. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 19
  4. 2025RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Drivingcited by 18
  5. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 14
  6. 2025CoLMDriver: LLM-based Negotiation Benefits Cooperative Autonomous Drivingcited by 13
  7. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Modelscited by 13
  8. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 12
  9. 2025DriveQA: Passing the Driving Knowledge Testcited by 10
  10. 2025ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Modelscited by 9
  11. 2025MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Casescited by 9
  12. 2026Accelerating Structured Chain-of-Thought in Autonomous Vehiclescited by 6
  13. 2025INSIGHT: Enhancing Autonomous Driving Safety through Vision-Language Models on Context-Aware Hazard Detection and Edge Case Evaluationcited by 5
  14. 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Drivingcited by 5
  15. 2025VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Processcited by 4
  16. 2025SafeMT: Multi-turn Safety for Multimodal Language Modelscited by 4
  17. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialoguescited by 4
  18. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 3
  19. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  20. 2025ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Expertscited by 3
  21. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  22. 2025Embodied4C: Measuring What Matters for Embodied Vision-Language Navigationcited by 3
  23. 2026Physical Prompt Injection Attacks on Large Vision-Language Modelscited by 3
  24. 2026VLGA: Vision-Language-Geometry-Action Models for Autonomous Drivingcited by 3
  25. 2024From Imitation to Exploration: End-to-end Autonomous Driving based on World Modelcited by 2
  26. 2025TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videoscited by 2
  27. 2025Bridging Human Oversight and Black-box Driver Assistance: Vision-Language Models for Predictive Alerting in Lane Keeping Assist Systemscited by 2
  28. 2025Safety2Drive: Safety-Critical Scenario Benchmark for the Evaluation of Autonomous Drivingcited by 2
  29. 2026Listen, Look, Drive: Coupling Audio Instructions for User-aware VLA-based Autonomous Drivingcited by 2
  30. 2025LaViPlan : Language-Guided Visual Path Planning with RLVRcited by 1
  31. 2026AlignDrive: Aligned Lateral-Longitudinal Planning for End-to-End Autonomous Drivingcited by 1
  32. 2026Risk-Aware World Model Predictive Control for Generalizable End-to-End Autonomous Drivingcited by 1
  33. 2026TaCarla: A comprehensive benchmarking dataset for end-to-end autonomous drivingcited by 1
  34. 2026DRIV-EX: Counterfactual Explanations for Driving LLMscited by 1
  35. 2026RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillationcited by 1
  36. 2026Teaching Vision-Language-Action Models What to See and Where to Lookcited by 1
  37. 2024LASER: Script Execution by Autonomous Agents for On-demand Traffic Simulationcited by 0
  38. 2025Robusto-1 Dataset: Comparing Humans and VLMs on real out-of-distribution Autonomous Driving VQA from Perucited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment
  2. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  3. 2025LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
  4. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
  5. 2026FROST-Drive: Scalable and Efficient End-to-End Driving with a Frozen Vision Encoder
  6. 2026Less Language, More Latents: Annotation-Efficient VLAs for Driving
  7. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios
  8. 2026Towards Reliable Vision-Language Models for Autonomous Driving