mlsanityData to 7 Oct 2026

LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs

arXiv 2312.04372 ↗First posted 7 Dec 2023cs.CL cs.AIcited by 45

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Autonomous driving (AD) has made significant strides in recent years. However, existing frameworks struggle to interpret and execute spontaneous user instructions, such as "overtake the car ahead." Large Language Models (LLMs) have demonstrated impressive reasoning capabilities showing potential to bridge this gap. In this paper, we present LaMPilot, a novel framework that integrates LLMs into AD systems, enabling them to follow user instructions by generating code that leverages established functional primitives. We also introduce LaMPilot-Bench, the first benchmark dataset specifically designed to quantitatively evaluate the efficacy of language model programs in AD. Adopting the LaMPilot framework, we conduct extensive experiments to assess the performance of off-the-shelf LLMs on LaMPilot-Bench. Our results demonstrate the potential of LLMs in handling diverse driving scenarios and following user instructions in driving. To facilitate further research in this area, we release our code and data at https://github.com/PurdueDigitalTwin/LaMPilot.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 56 of its 92 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

56 of 92 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspective28 Nov
  2. 2023A Survey on Multimodal Large Language Models for Autonomous Driving21 Nov
  3. 2023A Language Agent for Autonomous Driving17 Nov
  4. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Driving2 Nov
  5. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles12 Oct
  6. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving4 Oct
  7. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving3 Oct
  8. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  9. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models28 Sep
  10. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles19 Sep
  11. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2022Planning-oriented Autonomous Drivingcited by 1,068
33 more
  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  3. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  4. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  7. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  10. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  11. 2023GPT-Driver: Learning to Drive with GPTcited by 344
  12. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  13. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  14. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  15. 2019Talk2Car: Taking Control of Your Self-Driving Carcited by 141
  16. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  17. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  18. 2017On a Formal Model of Safe and Scalable Self-driving Carscited by 129
  19. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 122
  20. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  21. 2023ReasonNet: End-to-End Driving with Temporal and Global Reasoningcited by 92
  22. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  23. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  24. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  25. 2023ADAPT: Action-aware Driving Caption Transformercited by 78
  26. 2017Learning with Latent Languagecited by 75
  27. 2022Code4Struct: Code Generation for Few-Shot Event Structure Predictioncited by 70
  28. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  29. 2023Interactive Natural Language Processingcited by 31
  30. 2023CodeFusion: A Pre-trained Diffusion Model for Code Generationcited by 31
  31. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  32. 2023LeTI: Learning to Generate from Textual Interactionscited by 25
  33. 2023Human-Centric Autonomous Systems With LLMs for User Command Reasoningcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 18; 2025: 16; 2026: 10.

45 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  2. 2025SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignmentcited by 149
  3. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  4. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  5. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  6. 2024A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlookcited by 65
  7. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 46
  8. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  9. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  10. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  11. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  12. 2024OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 26
33 more
  1. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  2. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  3. 2024VLM-MPC: Vision Language Foundation Model (VLM)-Guided Model Predictive Controller (MPC) for Autonomous Drivingcited by 22
  4. 2024On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validationcited by 15
  5. 2024Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Drivingcited by 11
  6. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  7. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  8. 2024Generating and Evolving Reward Functions for Highway Driving with Large Language Modelscited by 8
  9. 2024LORD: Large Models based Opposite Reward Design for Autonomous Drivingcited by 6
  10. 2024CodeGraph: Enhancing Graph Reasoning of LLMs with Codecited by 6
  11. 2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
  12. 2024From Words to Wheels: Automated Style-Customized Policy Generation for Autonomous Drivingcited by 5
  13. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planningcited by 5
  14. 2025HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Drivingcited by 4
  15. 2026Drive My Way: Preference Alignment of Vision-Language-Action Model for Personalized Drivingcited by 4
  16. 2024A Survey on Large Language Model-empowered Autonomous Drivingcited by 3
  17. 2025Advancing Autonomous Vehicle Intelligence: Deep Learning and Multimodal LLM for Traffic Sign Recognition and Robust Lane Detectioncited by 3
  18. 2025On Simulation-Guided LLM-based Code Generation for Safe Autonomous Driving Softwarecited by 3
  19. 2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
  20. 2026How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Studycited by 2
  21. 2025Box-QAymo: Box-Referring VQA Dataset for Autonomous Drivingcited by 1
  22. 2025KEPT: Knowledge-Enhanced Prediction of Trajectories from Consecutive Driving Frames with Vision-Language Modelscited by 1
  23. 2026Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planningcited by 1
  24. 2026Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAscited by 1
  25. 2024LASMP: Language Aided Subset Sampling Based Motion Plannercited by 0
  26. 2025Human-like Semantic Navigation for Autonomous Driving using Knowledge Representation and Large Language Modelscited by 0
  27. 2025Scene-Aware Conversational ADAS with Generative AI for Real-Time Driver Assistancecited by 0
  28. 2026UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Drivingcited by 0
  29. 2026Large Multimodal Agents for Intelligent Transportation Systems: Architectures, Evidence, and Deployment Challengescited by 0
  30. 2026NARRATE: A Multimodal Real-World Australian Driving Dataset for Human-Centred Explanations in Automated Drivingcited by 0
  31. 2026FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Modelscited by 0
  32. 2026doPlan: A Variable-Horizon Dataset for Multi-Stage Language-Conditioned Planning in Autonomous Drivingcited by 0
  33. 2026DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models for Human-like Autonomous Driving: A Survey
  2. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving
  3. 2025LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
  4. 2026Rethinking Language's Role in Efficient VLA for Autonomous Vehicles: Toward Smarter, Trustworthy Driving
  5. 2025AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving
  6. 2023Vision Language Models in Autonomous Driving: A Survey and Outlook
  7. 2025A Survey on Vision-Language-Action Models for Autonomous Driving
  8. 2025AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving