mlsanityData to 7 Oct 2026

XLM for Autonomous Driving Systems: A Comprehensive Review

arXiv 2409.10484 ↗First posted 16 Sep 2024eess.SY cs.SYcited by 5

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Large Language Models (LLMs) have showcased remarkable proficiency in various information-processing tasks. These tasks span from extracting data and summarizing literature to generating content, predictive modeling, decision-making, and system controls. Moreover, Vision Large Models (VLMs) and Multimodal LLMs (MLLMs), which represent the next generation of language models, a.k.a., XLMs, can combine and integrate many data modalities with the strength of language understanding, thus advancing several information-based systems, such as Autonomous Driving Systems (ADS). Indeed, by combining language communication with multimodal sensory inputs, e.g., panoramic images and LiDAR or radar data, accurate driving actions can be taken. In this context, we provide in this survey paper a comprehensive overview of the potential of XLMs towards achieving autonomous driving. Specifically, we review the relevant literature on ADS and XLMs, including their architectures, tools, and frameworks. Then, we detail the proposed approaches to deploy XLMs for autonomous driving solutions. Finally, we provide the related challenges to XLM deployment for ADS and point to future research directions aiming to enable XLM adoption in future ADS frameworks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 101 of its 141 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

101 of 141 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Models7 May
  2. 2024Engineering Safety Requirements for Autonomous Driving with Large Language Models24 Mar
  3. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving21 Feb
  4. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model16 Feb
  5. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Driving2 Feb
  6. 2024A Survey for Foundation Models in Autonomous Driving2 Feb
  7. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving14 Dec
  8. 2023Evaluation of Large Language Models for Decision Making in Autonomous Driving11 Dec
  9. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspective28 Nov
  10. 2023A Survey on Multimodal Large Language Models for Autonomous Driving21 Nov
  11. 2023Applications of Large Scale Foundation Models for Autonomous Driving20 Nov
  12. 2023A Language Agent for Autonomous Driving17 Nov
6 more
  1. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving2 Nov
  2. 2023Dialogue-based generation of self-driving simulation scenarios using Large Language Models26 Oct
  3. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  4. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  6. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep

Other references · 83

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
38 more
  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  13. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  14. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  15. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  16. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  17. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  18. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  19. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  20. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  21. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  22. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  23. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  24. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  25. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  26. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  27. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  28. 2023Multimodal Large Language Models: A Surveycited by 193
  29. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  30. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  31. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  32. 2022DRAMA: Joint Risk Localization and Captioning in Drivingcited by 125
  33. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  34. 2023Language Prompt for Autonomous Drivingcited by 110
  35. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  36. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  37. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  38. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  2. 2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
  3. 2025ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Modelscited by 8
  4. 2025CleanMAP: Distilling Multimodal LLMs for Confidence-Driven Crowdsourced HD Map Updatescited by 1
  5. 2024A Novel MLLM-based Approach for Autonomous Driving in Different Weather Conditionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  2. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  3. 2024Probing Multimodal LLMs as World Models for Driving
  4. 2025BetterCheck: Towards Safeguarding VLMs for Automotive Perception Systems
  5. 2024Application of Multimodal Large Language Models in Autonomous Driving
  6. 2024A Superalignment Framework in Autonomous Driving with Large Language Models
  7. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
  8. 2024Generalizing End-To-End Autonomous Driving In Real-World Environments Using Zero-Shot LLMs