mlsanityData to 7 Oct 2026

Applications of Large Scale Foundation Models for Autonomous Driving

arXiv 2311.12144 ↗First posted 20 Nov 2023cs.CVcited by 14

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Since DARPA Grand Challenges (rural) in 2004/05 and Urban Challenges in 2007, autonomous driving has been the most active field of AI applications. Recently powered by large language models (LLMs), chat systems, such as chatGPT and PaLM, emerge and rapidly become a promising direction to achieve artificial general intelligence (AGI) in natural language processing (NLP). There comes a natural thinking that we could employ these abilities to reformulate autonomous driving. By combining LLM with foundation models, it is possible to utilize the human knowledge, commonsense and reasoning to rebuild autonomous driving systems from the current long-tailed AI dilemma. In this paper, we investigate the techniques of foundation models and LLMs applied for autonomous driving, categorized as simulation, world model, data annotation and planning or E2E solutions etc.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 214 of its 248 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

214 of 248 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023A Language Agent for Autonomous Driving17 Nov
  2. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving2 Nov
  3. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  4. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  5. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  6. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  7. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles19 Sep
  8. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 206

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  13. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  14. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  15. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  16. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  17. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  18. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  19. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  20. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  21. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  22. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  23. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  24. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  25. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  26. 2023A Survey of Large Language Modelscited by 2,537
  27. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  28. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  29. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  30. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  31. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  32. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  33. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  34. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  35. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  36. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  37. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  38. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422

50 most cited of 206 All 206 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 9; 2025: 5.

14 papers.

Sort:

By area:

  1. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  2. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  3. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  4. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  5. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  6. 2024CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Masterycited by 30
  7. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  8. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  9. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilitiescited by 6
  10. 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
  11. 2024Evaluating and Enhancing Trustworthiness of LLMs in Perception Taskscited by 4
  12. 2024How Could Generative AI Support Compliance with the EU AI Act? A Review for Safe Automated Driving Perceptioncited by 2
2 more
  1. 2025Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptationcited by 0
  2. 2025Benchmarking Autonomous Driving Planners Across Leaderboards: A Unified CARLA-Based Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  2. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
  3. 2024Testing Large Language Models on Driving Theory Knowledge and Skills for Connected Autonomous Vehicles
  4. 2023A Survey on Multimodal Large Language Models for Autonomous Driving
  5. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving
  6. 2024A Survey on Large Language Model-empowered Autonomous Driving
  7. 2025Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances
  8. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving