# Human-Centric Autonomous Systems With LLMs for User Command Reasoning arXiv: https://arxiv.org/abs/2311.08206 ยท HTML: https://mlsanity.com/paper/2311.08206/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning. - Builds on 14 of its 46 references that are in the map. - First posted on arXiv 14 Nov 2023. ## Placed in - LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/) ## Builds on (most cited first, 14 of 14) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021) - [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022) - [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023) - [Drive Like a Human: Rethinking Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2307.07162/) (2023) - [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023) - [Language Prompt for Autonomous Driving](https://mlsanity.com/paper/2309.04379/) (2023) - [Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2309.10228/) (2023) - [ADAPT: Action-aware Driving Caption Transformer](https://mlsanity.com/paper/2302.00673/) (2023) - [Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2310.08034/) (2023) - [HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Driving](https://mlsanity.com/paper/2309.05186/) (2023) ## Built on by (most cited first, 22 of 22) - [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023) - [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024) - [A Survey for Foundation Models in Autonomous Driving](https://mlsanity.com/paper/2402.01105/) (2024) - [LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs](https://mlsanity.com/paper/2312.04372/) (2023) - [Probing Multimodal LLMs as World Models for Driving](https://mlsanity.com/paper/2405.05956/) (2024) - [Towards Human-Centric Autonomous Driving: A Fast-Slow Architecture Integrating Large Language Model Guidance with Reinforcement Learning](https://mlsanity.com/paper/2505.06875/) (2025) - [Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models](https://mlsanity.com/paper/2604.01259/) (2026) - [From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysis](https://mlsanity.com/paper/2502.11919/) (2025) - [A Survey on Human-AI Collaboration with Large Foundation Models](https://mlsanity.com/paper/2403.04931/) (2024) - [XLM for Autonomous Driving Systems: A Comprehensive Review](https://mlsanity.com/paper/2409.10484/) (2024) - [From Words to Wheels: Automated Style-Customized Policy Generation for Autonomous Driving](https://mlsanity.com/paper/2409.11694/) (2024) - [Hard Cases Detection in Motion Prediction by Vision-Language Foundation Models](https://mlsanity.com/paper/2405.20991/) (2024) - [A Survey on Large Language Model-empowered Autonomous Driving](https://mlsanity.com/paper/2409.14165/) (2024) - [TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models](https://mlsanity.com/paper/2412.15462/) (2024) - [Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges](https://mlsanity.com/paper/2512.00021/) (2025) - [JMI at SemEval 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models](https://mlsanity.com/paper/2403.04798/) (2024) - [Sage Deer: A Super-Aligned Driving Generalist Is Your Copilot](https://mlsanity.com/paper/2505.10257/) (2025) - [E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving](https://mlsanity.com/paper/2512.04733/) (2025) - [Traj-LLM: A New Exploration for Empowering Trajectory Prediction with Pre-trained Large Language Models](https://mlsanity.com/paper/2405.04909/) (2024) - [A Large-Language-Model Supported Personalized Driving Framework for Lane Change in Highway Scenarios](https://mlsanity.com/paper/2606.31483/) (2026) - [Talk to Me, Jarvis: An Open-Source Edge-Deployable Voice Assistant Framework for Autonomous Racecars](https://mlsanity.com/paper/2609.21109/) (2026) - [A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform](https://mlsanity.com/paper/2610.00926/) (2026) ## Abstract (arXiv) The evolution of autonomous driving has made remarkable advancements in recent years, evolving into a tangible reality. However, a human-centric large-scale adoption hinges on meeting a variety of multifaceted requirements. To ensure that the autonomous system meets the user's intent, it is essential to accurately discern and interpret user commands, especially in complex or emergency situations. To this end, we propose to leverage the reasoning capabilities of Large Language Models (LLMs) to infer system requirements from in-cabin users' commands. Through a series of experiments that include different LLM models and prompt designs, we explore the few-shot multivariate binary classification accuracy of system requirements from natural language textual commands. We confirm the general ability of LLMs to understand and reason about prompts but underline that their effectiveness is conditioned on the quality of both the LLM model and the design of appropriate sequential prompts. Code and models are public with the link urlhttps://github.com/KTH-RPL/DriveCmdLLM.