Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
On the map: End-to-end autonomous driving → Reasoning VLA and VLM driving policies
Abstract (arXiv)
The emergence of multi-modal foundation models has markedly transformed the technology for autonomous driving, shifting away from conventional and mostly hand-crafted design choices towards unified, foundation-model-based approaches, capable of directly inferring motion trajectories from raw sensory inputs. This new class of methods can also incorporate natural language as an additional modality, with Vision-Language-Action (VLA) models serving as a representative example. In this review, we provide a comprehensive examination of such methods through a unifying taxonomy to critically evaluate their architectural design choices, methodological strengths, and their inherent capabilities and limitations. Our survey covers 37 recently proposed approaches that span the landscape of trajectory planning with foundation models. Furthermore, we assess these approaches with respect to the openness of their source code and datasets, offering valuable information to practitioners and researchers. We provide an accompanying webpage that catalogues the methods based on our taxonomy, available at: https://github.com/fiveai/FMs-for-driving-trajectories
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning VLA and VLM driving policies, one of 9 directions in End-to-end autonomous driving. Builds on 152 of its 183 references that are in the map. First posted on arXiv 31 Oct 2025.
Builds on
152 of 183 references.
Closest ancestors · 19
Same direction, newest first
- 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail30 Oct
- 2025A Survey on Vision-Language-Action Models for Autonomous Driving30 Jun
- 2025AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning16 Jun
- 2025S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation30 May
- 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models29 May
- 2025Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects26 May
- 2025DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving26 May
- 2025SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving22 May
- 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving21 May
- 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning6 Apr
- 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Model30 Mar
- 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation25 Mar
7 more
- 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment12 Mar
- 2025FASIONAD++ : Integrating High-Level Instruction and Information Bottleneck in FAt-Slow fusION Systems for Enhanced Safety in Autonomous Driving with Adaptive Feedback11 Mar
- 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning10 Mar
- 2025Distilling Multi-modal Large Language Models for Autonomous Driving16 Jan
- 2024Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving10 Sep
- 2024VLP: Vision Language Planning for Autonomous Driving10 Jan
- 2023Towards Knowledge-driven Autonomous Driving7 Dec
Other references · 133
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023Visual Instruction Tuningcited by 7,792
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
- 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
- 2022Planning-oriented Autonomous Drivingcited by 1,068
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
- 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
- 2021Domain Generalization: A Surveycited by 760
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
- 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
- 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
- 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
- 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
- 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
- 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
- 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
- 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
- 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
- 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
- 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
- 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
50 most cited of 133 All 133 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
- 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future
- 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis
- 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities
- 2025DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models
- 2024LeGo-Drive: Language-enhanced Goal-oriented Closed-Loop End-to-End Autonomous Driving
- 2026Unifying Language-Action Understanding and Generation for Autonomous Driving
- 2026Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Driving