# Driving Everywhere with Large Language Model Policy Adaptation

arXiv: https://arxiv.org/abs/2402.05932 · HTML: https://mlsanity.com/paper/2402.05932/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Placed in

- End-to-end autonomous driving / [Large language models for driving](https://mlsanity.com/driving/large-language-models-for-driving/)

## Builds on (most cited first, 22 of 22)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023)
- [Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?](https://mlsanity.com/paper/2006.14911/) (2020)
- [VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles](https://mlsanity.com/paper/2111.12083/) (2021)
- [Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models](https://mlsanity.com/paper/2310.17642/) (2023)
- [Semantic Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2305.11307/) (2023)
- [Interactive Task Planning with Language Models](https://mlsanity.com/paper/2310.10645/) (2023)
- [Plan Explicability and Predictability for Robot Task Planning](https://mlsanity.com/paper/1511.08158/) (2015)
- [Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning](https://mlsanity.com/paper/2209.11820/) (2022)
- [Interpretable Self-Aware Neural Networks for Robust Trajectory Prediction](https://mlsanity.com/paper/2211.08701/) (2022)
- [Rule-based Optimal Control for Autonomous Driving](https://mlsanity.com/paper/2101.05709/) (2021)
- [Robustness to Out-of-Distribution Inputs via Task-Aware Generative Uncertainty](https://mlsanity.com/paper/1812.10687/) (2018)
- [Formalizing Traffic Rules for Machine Interpretability](https://mlsanity.com/paper/2007.00330/) (2020)
- [Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles](https://mlsanity.com/paper/2207.12380/) (2022)
- [Receding Horizon Planning with Rule Hierarchies for Autonomous Vehicles](https://mlsanity.com/paper/2212.03323/) (2022)
- [Multi-Predictor Fusion: Combining Learning-based and Rule-based Trajectory Predictors](https://mlsanity.com/paper/2307.01408/) (2023)

## Built on by (most cited first, 30 of 53)

- [CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models](https://mlsanity.com/paper/2503.22020/) (2025)
- [AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.13757/) (2025)
- [FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving](https://mlsanity.com/paper/2505.17685/) (2025)
- [Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving](https://mlsanity.com/paper/2407.00959/) (2024)
- [VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models](https://mlsanity.com/paper/2404.00210/) (2024)
- [PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning](https://mlsanity.com/paper/2406.01587/) (2024)
- [Safety Guardrails for LLM-Enabled Robots](https://mlsanity.com/paper/2503.07885/) (2025)
- [Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future](https://mlsanity.com/paper/2512.16760/) (2025)
- [Adversarial Attacks on Robotic Vision Language Action Models](https://mlsanity.com/paper/2506.03350/) (2025)
- [Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systems](https://mlsanity.com/paper/2405.20774/) (2024)
- [V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multimodal Large Language Models](https://mlsanity.com/paper/2502.09980/) (2025)
- [VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenes](https://mlsanity.com/paper/2405.05885/) (2024)
- [ROADWork: A Dataset and Benchmark for Learning to Recognize, Observe, Analyze and Drive Through Work Zones](https://mlsanity.com/paper/2406.07661/) (2024)
- [Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective](https://mlsanity.com/paper/2511.11478/) (2025)
- [AD-H: Language-guided Autonomous Driving with Hierarchical Agents](https://mlsanity.com/paper/2406.03474/) (2024)
- [Driving with Regulation: Trustworthy and Interpretable Decision-Making for Autonomous Driving with Retrieval-Augmented Reasoning](https://mlsanity.com/paper/2410.04759/) (2024)
- [Learning Vision-Language-Action World Models for Autonomous Driving](https://mlsanity.com/paper/2604.09059/) (2026)
- [LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends](https://mlsanity.com/paper/2410.15281/) (2024)
- [Distilling On-device Language Models for Robot Planning with Minimal Human Intervention](https://mlsanity.com/paper/2506.17486/) (2025)
- [DriveQA: Passing the Driving Knowledge Test](https://mlsanity.com/paper/2508.21824/) (2025)
- [UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving](https://mlsanity.com/paper/2512.09864/) (2025)
- [Unifying Language-Action Understanding and Generation for Autonomous Driving](https://mlsanity.com/paper/2603.01441/) (2026)
- [nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving](https://mlsanity.com/paper/2605.31572/) (2026)
- [Wolf: Dense Video Captioning with a World Summarization Framework](https://mlsanity.com/paper/2407.18908/) (2024)
- [Tracking Meets Large Multimodal Models for Driving Scenario Understanding](https://mlsanity.com/paper/2503.14498/) (2025)
- [Learning to Drive in New Cities Without Human Demonstrations](https://mlsanity.com/paper/2602.15891/) (2026)
- [Learning to Tune Like an Expert: Interpretable and Scene-Aware Navigation via MLLM Reasoning and CVAE-Based Adaptation](https://mlsanity.com/paper/2507.11001/) (2025)
- [TraveLLM: Could you plan my new public transit route in face of a network disruption?](https://mlsanity.com/paper/2407.14926/) (2024)
- [Planning Safety Trajectories with Dual-Phase, Physics-Informed, and Transportation Knowledge-Driven Large Language Models](https://mlsanity.com/paper/2504.04562/) (2025)
- [LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Driving](https://mlsanity.com/paper/2507.05754/) (2025)

## Abstract (arXiv)

Adapting driving behavior to new environments, customs, and laws is a long-standing problem in autonomous driving, precluding the widespread deployment of autonomous vehicles (AVs). In this paper, we present LLaDA, a simple yet powerful tool that enables human drivers and autonomous vehicles alike to drive everywhere by adapting their tasks and motion plans to traffic rules in new locations. LLaDA achieves this by leveraging the impressive zero-shot generalizability of large language models (LLMs) in interpreting the traffic rules in the local driver handbook. Through an extensive user study, we show that LLaDA's instructions are useful in disambiguating in-the-wild unexpected situations. We also demonstrate LLaDA's ability to adapt AV motion planning policies in real-world datasets; LLaDA outperforms baseline planning approaches on all our metrics. Please check our website for more details: https://boyiliee.github.io/llada.