mlsanityData to 7 Oct 2026

Distilling Multi-modal Large Language Models for Autonomous Driving

arXiv 2501.09757 ↗First posted 16 Jan 2025cs.CV cs.ROcited by 46

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

also in 1 more
Abstract (arXiv)

Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 57 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

52 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models19 Feb
  2. 2023DriveLM: Driving with Graph Visual Question Answering21 Dec
  3. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  4. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  12. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
36 more
  1. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  2. 2022Planning-oriented Autonomous Drivingcited by 1,068
  3. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  4. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  5. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  6. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  7. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  8. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  9. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  10. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  11. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  12. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  13. 2022Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baselinecited by 247
  14. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  15. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  16. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  17. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  18. 2023Scene as Occupancycited by 208
  19. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  20. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  21. 2022BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Drivingcited by 191
  22. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  23. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  24. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  25. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  26. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  27. 2022Learning from All Vehiclescited by 135
  28. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  29. 2024VLP: Vision Language Planning for Autonomous Drivingcited by 125
  30. 2023Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoningcited by 63
  31. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Drivingcited by 55
  32. 2023MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Unsignalized Intersectionscited by 38
  33. 2022Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Drivingcited by 30
  34. 2024MV2DFusion: Leveraging Modality-Specific Object Semantics for Multi-Modal 3D Detectioncited by 23
  35. 2023Look, Remember and Reason: Grounded reasoning in videos with language modelscited by 13
  36. 2024FutureDepth: Learning to Predict the Future Improves Video Depth Estimationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 26; 2026: 20.

46 papers.

Sort:

By area:

  1. 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuningcited by 252
  2. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  3. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 85
  4. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  5. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  6. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  7. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  8. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  9. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Drivingcited by 27
  10. 2025ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Drivingcited by 22
  11. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  12. 2025PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Drivingcited by 16
34 more
  1. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scalecited by 12
  2. 2026EponaV2: Driving World Model with Comprehensive Future Reasoningcited by 12
  3. 2025Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Drivingcited by 11
  4. 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agentscited by 11
  5. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarioscited by 11
  6. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  7. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  8. 2025RoCA: Robust Cross-Domain End-to-End Autonomous Drivingcited by 8
  9. 2025CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refinecited by 7
  10. 2025Efficient Multi-Camera Tokenization with Triplanes for End-to-End Drivingcited by 6
  11. 2025GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Drivingcited by 6
  12. 2026HERMES: A Holistic End-to-End Risk-Aware Multimodal Embodied System with Vision-Language Models for Long-Tail Autonomous Drivingcited by 5
  13. 2025CogAD: Cognitive-Hierarchy Guided End-to-End Autonomous Drivingcited by 4
  14. 2026NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Drivingcited by 4
  15. 2026HAD: Combining Hierarchical Diffusion with Metric-Decoupled RL for End-to-End Drivingcited by 4
  16. 2025Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Surveycited by 3
  17. 2025More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Modelscited by 3
  18. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  19. 2026EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillationcited by 3
  20. 2025AccidentSim: Generating Vehicle Collision Videos with Physically Realistic Collision Trajectories from Real-World Accident Reportscited by 2
  21. 2026Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Drivingcited by 2
  22. 2025KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Modelscited by 1
  23. 2025Prompt-Driven Domain Adaptation for End-to-End Autonomous Driving via In-Context RLcited by 1
  24. 2025LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Drivingcited by 1
  25. 2026Modular Autonomy with Conversational Interaction: An LLM-driven Framework for Decision Making in Autonomous Drivingcited by 1
  26. 2026VL-DPO: Vision-Language-Guided Finetuning for Preference-Aligned Autonomous Drivingcited by 1
  27. 2026VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Drivingcited by 1
  28. 2026Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAscited by 1
  29. 2026Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Modelscited by 0
  30. 2026Cross-Stage Coherence in Hierarchical Driving VQA: Explicit Baselines and Learned Gated Context Projectorscited by 0
  31. 2026Steins;Gate Drive: Semantic Safety Arbitration over Structured Futures for Latency-Decoupled LLM Planningcited by 0
  32. 2026Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddingscited by 0
  33. 2026FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memorycited by 0
  34. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
  2. 2024Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving
  3. 2025DeeAD: Dynamic Early Exit of Vision-Language Action for Efficient Autonomous Driving
  4. 2025VLMPlanner: Integrating Visual Language Models with Motion Planning
  5. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Driving
  6. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs
  7. 2026Less Language, More Latents: Annotation-Efficient VLAs for Driving
  8. 2025ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving