mlsanityData to 7 Oct 2026

Agent-driven Long-tail Simulation for Autonomous Driving

arXiv 2607.04331 ↗First posted 5 Jul 2026cs.RO cs.CVcited by 0

On the map: End-to-end autonomous driving → Large language models for driving

also in 1 more
Abstract (arXiv)

Evaluating autonomous driving systems in closed-loop settings requires realistic and interactive simulation, yet existing simulators largely rely on log replay or rule-based agents, limiting behavioral diversity and long-tail coverage. We propose an agent-driven simulation framework in which surrounding road participants are controlled by instruction-following large language models through a structured action interface, enabling intentional and reactive behaviors while preserving physical plausibility. Furthermore, we introduce SemanticPlan, a benchmark of closed-loop planning in long-tail and semantically rich scenarios that augment real nuPlan scenes with multiple interactive agents following diverse language instructions. Evaluation results show that state-of-the-art planners still struggle to consistently achieve safe and effective task completion, suggesting that these long-tail scenarios remain challenging.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 50 of its 58 references that are in the map. First posted on arXiv 5 Jul 2026.

Builds on

50 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving5 Jul
  2. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Driving20 Jun

Other references · 48

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  4. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  5. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  6. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  7. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  8. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  9. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 158
  10. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  11. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  12. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 138
36 more
  1. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
  2. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  3. 2023V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecastingcited by 114
  4. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  5. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 103
  6. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  7. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  8. 2023The Waymo Open Sim Agents Challengecited by 86
  9. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  10. 2023ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modelingcited by 60
  11. 2025WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarioscited by 60
  12. 2023LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoningcited by 59
  13. 2023CAT: Closed-loop Adversarial Training for Safe End-to-End Drivingcited by 52
  14. 2023From Prediction to Planning With Goal Conditioned Lane Graph Traversalscited by 45
  15. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  16. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  17. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  18. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  19. 2024DriveGPT: Scaling Autoregressive Behavior Models for Drivingcited by 29
  20. 2025CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulationcited by 28
  21. 2024Reasoning Multi-Agent Behavioral Topology for Interactive Autonomous Drivingcited by 27
  22. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 27
  23. 2023Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Drivingcited by 26
  24. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 26
  25. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  26. 2025Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modelingcited by 22
  27. 2024Making Large Language Models into World Models with Precondition and Effect Knowledgecited by 21
  28. 2024Promptable Closed-loop Traffic Simulationcited by 20
  29. 2025SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Modelcited by 20
  30. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  31. 2023Interactive Joint Planning for Autonomous Vehiclescited by 13
  32. 2025TrajTok: Technical Report for 2025 Waymo Open Sim Agents Challengecited by 8
  33. 2025CrashAgent: Crash Scenario Generation via Multi-modal Reasoningcited by 5
  34. 2025DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulationcited by 5
  35. 2024Adversarial and Reactive Traffic Entities for Behavior-Realistic Driving Simulation: A Reviewcited by 3
  36. 2025nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous Driving via Reactive Multi-Agent Simulationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023A Language Agent for Autonomous Driving
  2. 2024AD-H: Language-guided Autonomous Driving with Hierarchical Agents
  3. 2026Large Language Model based Interactive Decision-Making for Autonomous Driving
  4. 2026Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges
  5. 2026Bridging Large-Model Reasoning and Real-Time Control via Agentic Fast-Slow Planning
  6. 2026Multi-Agent Embodied Autonomous Driving (MAEAD): From V2X Information Exchange to Shared World Models
  7. 2025Align2Act: Instruction-Tuned Models for Human-Aligned Autonomous Driving
  8. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios