# SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Data

arXiv: https://arxiv.org/abs/2309.13193 · HTML: https://mlsanity.com/paper/2309.13193/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning.
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 22 Sep 2023.

## Placed in

- LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Drive Like a Human: Rethinking Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2307.07162/) (2023)
- [Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2309.10228/) (2023)
- [Embodied Task Planning with Large Language Models](https://mlsanity.com/paper/2307.01848/) (2023)
- [Dr.ICL: Demonstration-Retrieved In-context Learning](https://mlsanity.com/paper/2305.14128/) (2023)
- [A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models](https://mlsanity.com/paper/2308.07997/) (2023)
- [SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments](https://mlsanity.com/paper/2309.04077/) (2023)
- [Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents](https://mlsanity.com/paper/2305.02412/) (2023)

## Built on by (most cited first, 24 of 24)

- [Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trends](https://mlsanity.com/paper/2409.14457/) (2024)
- [Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances](https://mlsanity.com/paper/2502.16804/) (2025)
- [DebFlow: Automating Agent Creation via Agent Debate](https://mlsanity.com/paper/2503.23781/) (2025)
- [WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving](https://mlsanity.com/paper/2511.20022/) (2025)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarios](https://mlsanity.com/paper/2505.17209/) (2025)
- [A Survey on Large Language Model-empowered Autonomous Driving](https://mlsanity.com/paper/2409.14165/) (2024)
- [QuadSentinel: Sequent Safety for Machine-Checkable Control in Multi-agent Systems](https://mlsanity.com/paper/2512.16279/) (2025)
- [Carbon and Silicon, Coexist or Compete? A Survey on Human-AI Interactions in Agent-based Modeling and Simulation](https://mlsanity.com/paper/2502.18145/) (2025)
- [Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges](https://mlsanity.com/paper/2512.00021/) (2025)
- [CIA: Inferring the Communication Topology from LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2604.12461/) (2026)
- [GeoVLM: Improving Automated Vehicle Geolocalisation Using Vision-Language Matching](https://mlsanity.com/paper/2505.13669/) (2025)
- [Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management](https://mlsanity.com/paper/2510.03310/) (2025)
- [Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulations](https://mlsanity.com/paper/2510.13982/) (2025)
- [Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends](https://mlsanity.com/paper/2504.16134/) (2025)
- [A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach](https://mlsanity.com/paper/2512.11944/) (2025)
- [A Novel MLLM-based Approach for Autonomous Driving in Different Weather Conditions](https://mlsanity.com/paper/2411.10603/) (2024)
- [LimSim Series: An Autonomous Driving Simulation Platform for Validation and Enhancement](https://mlsanity.com/paper/2502.09170/) (2025)
- [Sky-Drive: A Distributed Multi-Agent Simulation Platform for Human-AI Collaborative and Socially-Aware Future Transportation](https://mlsanity.com/paper/2504.18010/) (2025)
- [Understanding Driving Risks using Large Language Models: Toward Elderly Driver Assessment](https://mlsanity.com/paper/2507.08367/) (2025)
- [SocialDriveGen: Generating Diverse Traffic Scenarios with Controllable Social Interactions](https://mlsanity.com/paper/2512.01363/) (2025)
- [Language-Driven Cost Optimization for Autonomous Driving](https://mlsanity.com/paper/2606.10974/) (2026)
- [Vision-Language Assistant for Emotional Reactions to Risky Driving](https://mlsanity.com/paper/2607.16181/) (2026)
- [Are LLMs Good Financial User Simulators? Multi-view Investor Logic Alignment (MILA)](https://mlsanity.com/paper/2609.15727/) (2026)

## Abstract (arXiv)

Leveraging advanced reasoning capabilities and extensive world knowledge of large language models (LLMs) to construct generative agents for solving complex real-world problems is a major trend. However, LLMs inherently lack embodiment as humans, resulting in suboptimal performance in many embodied decision-making tasks. In this paper, we introduce a framework for building human-like generative driving agents using post-driving self-report driving-thinking data from human drivers as both demonstration and feedback. To capture high-quality, natural language data from drivers, we conducted urban driving experiments, recording drivers' verbalized thoughts under various conditions to serve as chain-of-thought prompts and demonstration examples for the LLM-Agent. The framework's effectiveness was evaluated through simulations and human assessments. Results indicate that incorporating expert demonstration data significantly reduced collision rates by 81.04% and increased human likeness by 50% compared to a baseline LLM-based agent. Our study provides insights into using natural language-based human demonstration data for embodied tasks. The driving-thinking dataset is available at urlhttps://github.com/AIR-DISCOVER/Driving-Thinking-Dataset.