mlsanityData to 7 Oct 2026

A Survey for Foundation Models in Autonomous Driving

arXiv 2402.01105 ↗First posted 2 Feb 2024cs.LG cs.CV cs.ROcited by 51

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

The advent of foundation models has revolutionized the fields of natural language processing and computer vision, paving the way for their application in autonomous driving (AD). This survey presents a comprehensive review of more than 40 research papers, demonstrating the role of foundation models in enhancing AD. Large language models contribute to planning and simulation in AD, particularly through their proficiency in reasoning, code generation and translation. In parallel, vision foundation models are increasingly adapted for critical tasks such as 3D object detection and tracking, as well as creating realistic driving scenarios for simulation and testing. Multi-modal foundation models, integrating diverse inputs, exhibit exceptional visual understanding and spatial reasoning, crucial for end-to-end AD. This survey not only provides a structured taxonomy, categorizing foundation models based on their modalities and functionalities within the AD domain but also delves into the methods employed in current research. It identifies the gaps between existing foundation models and cutting-edge AD approaches, thereby charting future research directions and proposing a roadmap for bridging these gaps.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 58 of its 81 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

58 of 81 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Applications of Large Scale Foundation Models for Autonomous Driving20 Nov
  2. 2023A Language Agent for Autonomous Driving17 Nov
  3. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving2 Nov
  4. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles12 Oct
  5. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  6. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  7. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  8. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep

Other references · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2023Segment Anythingcited by 8,158
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  13. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  14. 2022Planning-oriented Autonomous Drivingcited by 1,068
  15. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  16. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  17. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  18. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  19. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  20. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  21. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  22. 2018Textual Explanations for Self-Driving Vehiclescited by 226
  23. 2023Segment and Track Anythingcited by 193
  24. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  25. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  26. 2023Real-Time Flying Object Detection with YOLOv8cited by 93
  27. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  28. 2023Language Conditioned Traffic Generationcited by 86
  29. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  30. 2023SAM3D: Zero-Shot 3D Object Detection via Segment Anything Modelcited by 40
  31. 2023Self-Supervised Image-to-Point Distillation via Semantically Tolerant Contrastive Losscited by 35
  32. 2023Can you text what is happening? Integrating pre-trained language encoders into trajectory prediction models for autonomous drivingcited by 30
  33. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22
  34. 2023Open-sourced Data Ecosystem in Autonomous Driving: the Present and Futurecited by 22
  35. 2023Learning to Adapt SAM for Segmenting Cross-domain Point Cloudscited by 20
  36. 2023Explaining Autonomous Driving Actions with Visual Question Answeringcited by 16
  37. 2023Robustness of Segment Anything Model (SAM) for Autonomous Driving in Adverse Weather Conditionscited by 14
  38. 2022Incorporating Voice Instructions in Model-Based Reinforcement Learning for Self-Driving Carscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 23; 2026: 17.

51 papers.

Sort:

By area:

  1. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  2. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  3. 2025A Survey on Vision-Language-Action Models for Autonomous Drivingcited by 72
  4. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  5. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  6. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  7. 2025Can Multimodal Large Language Models Understand Spatial Relations?cited by 20
  8. 2024FastSAM3D: An Efficient Segment Anything Model for 3D Volumetric Medical Imagescited by 14
  9. 2024MetaToken: Detecting Hallucination in Image Descriptions by Meta Classificationcited by 13
  10. 2025Reasoning Segmentation for Images and Videos: A Surveycited by 12
  11. 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospectscited by 12
  12. 2025Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoningcited by 10
38 more
  1. 2025SoccerMaster: A Vision Foundation Model for Soccer Understandingcited by 10
  2. 2024Utilizing Large Language Models for Information Extraction from Real Estate Transactionscited by 7
  3. 2024Exploring the Causality of End-to-End Autonomous Drivingcited by 7
  4. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  5. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluationcited by 7
  6. 2025Efficient Multi-Camera Tokenization with Triplanes for End-to-End Drivingcited by 6
  7. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilitiescited by 6
  8. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  9. 2025TS-VLM: Text-Guided SoftSort Pooling for Vision-Language Models in Multi-View Driving Reasoningcited by 5
  10. 2026StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Drivingcited by 5
  11. 2024Uncertainty-Guided Enhancement on Driving Perception System via Foundation Modelscited by 3
  12. 2025Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Drivingcited by 3
  13. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  14. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  15. 2025Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Drivingcited by 2
  16. 2025Can the Waymo Open Motion Dataset Support Realistic Behavioral Modeling? A Validation Study with Naturalistic Trajectoriescited by 2
  17. 2025Towards Efficient and Effective Multi-Camera Encoding for End-to-End Drivingcited by 2
  18. 2026Post-Training in End-to-End Autonomous Drivingcited by 2
  19. 2024Uniform Transformation: Refining Latent Representation in Variational Autoencoderscited by 1
  20. 2025Online Location Planning for AI-Defined Vehicles: Optimizing Joint Tasks of Order Serving and Spatio-Temporal Heterogeneous Model Fine-Tuningcited by 1
  21. 2025Efficient Contrastive Decoding with Probabilistic Hallucination Detection - Mitigating Hallucinations in Large Vision Language Models -cited by 1
  22. 2025GeoVLM: Improving Automated Vehicle Geolocalisation Using Vision-Language Matchingcited by 1
  23. 2026Position: Certifiable State Integrity Should Be Built from Local Validity, Not Global Scalecited by 1
  24. 2026Lying to Win: Assessing LLM Deception through Human-AI Games and Parallel-World Probingcited by 1
  25. 2025ECG-EmotionNet: Nested Mixture of Expert (NMoE) Adaptation of ECG-Foundation Model for Driver Emotion Recognitioncited by 0
  26. 2025Weakly Supervised Object Segmentation by Background Conditional Divergencecited by 0
  27. 2026Looking and Listening Inside and Outside: Multimodal Artificial Intelligence Systems for Driver Safety Assessment and Intelligent Vehicle Decision-Makingcited by 0
  28. 20261%>100%: High-Efficiency Visual Adapter with Complex Linear Projection Optimizationcited by 0
  29. 2026SCP: Spatial Causal Prediction in Videocited by 0
  30. 2026On the Strengths and Weaknesses of Data for Open-set Embodied Assistancecited by 0
  31. 2026Capability neq Interpretability: Human Interpretability of Vision Foundation Modelscited by 0
  32. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0
  33. 2026Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Modelscited by 0
  34. 2026Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddingscited by 0
  35. 2026Mirror Learningcited by 0
  36. 2026From Interpretability Methods to Interpretable Modelscited by 0
  37. 2026Planning in the Backbone: DiffAdapterVLA for Native Continuous Trajectory Generation with Driving VLMscited by 0
  38. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities
  2. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  3. 2025AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving
  4. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models
  5. 2023Foundation Models in Robotics: Applications, Challenges, and the Future
  6. 2025Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving
  7. 2023ADriver-I: A General World Model for Autonomous Driving
  8. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs