mlsanityData to 14 Oct 2026

Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Driving

arXiv 2607.10438 ↗First posted 11 Jul 2026cs.RO cs.NIcited by 0

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

Autonomous driving planning is a key component of IoT-enabled intelligent transportation systems, requiring vehicles to generate safe, efficient, and executable trajectories in complex urban environments from multi-source contextual information. While imitation learning (IL) has shown promise on large-scale datasets, IL-based planners still suffer from limited coverage of complex long-tail interactions, weak consistency with downstream constrained refinement, and insufficient use of high level scene semantics under real time constraints. To address these issues, this paper proposes a large language model (LLM) enhanced differentiable trajectory planning framework for IoT-enabled autonomous driving. Specifically, we introduce a surrounding agent centric data augmentation strategy to reorganize sur rounding agent trajectories as additional planning supervision, thereby improving the training distribution without collecting additional raw data. We further design a complexity-aware asyn chronous LLM-based semantic enhancement module to extract scene-related high-level semantic features with controlled online overhead. In addition, a differentiable optimization module is incorporated to refine generated trajectories with explicit residual penalties while backpropagating optimization gradients to the upstream planner. Experiments show that the proposed method achieves the best overall scores of 83.63 and 78.29 on the nuPlan closed-loop nonreactive and reactive Hard20 benchmarks, respectively, and CARLA-ROS tests further verify its online deployment and real time closed-loop execution capability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 34 of its 54 references that are in the map. First posted on arXiv 11 Jul 2026.

Builds on

34 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends20 Oct
  2. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Driving20 Jun
  3. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Models12 Dec

Other references · 31

  1. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  2. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  3. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  4. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  5. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 242
  6. 2018Differentiable MPC for End-to-end Planning and Controlcited by 189
  7. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  8. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 141
  9. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  10. 2023Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Drivingcited by 123
  11. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  12. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
19 more
  1. 2023Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunitiescited by 90
  2. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  3. 2021Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Drivingcited by 73
  4. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 73
  5. 2022Theseus: A Library for Differentiable Nonlinear Optimizationcited by 65
  6. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 62
  7. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  8. 2023From Prediction to Planning With Goal Conditioned Lane Graph Traversalscited by 45
  9. 2024Hierarchical Split Federated Learning: Convergence Analysis and System Optimizationcited by 38
  10. 2023Optimal Resource Allocation for U-Shaped Parallel Split Learningcited by 33
  11. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 31
  12. 2024IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengerscited by 20
  13. 2022Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curriculacited by 17
  14. 2024LeGo-Drive: Language-enhanced Goal-oriented Closed-Loop End-to-End Autonomous Drivingcited by 12
  15. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  16. 2025LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systemscited by 8
  17. 2025Robust Deep Reinforcement Learning in Robotics via Adaptive Gradient-Masked Adversarial Attackscited by 7
  18. 2025LEED: A Highly Efficient and Scalable LLM-Empowered Expert Demonstrations Framework for Multi-Agent Reinforcement Learningcited by 6
  19. 2024Augmenting Safety-Critical Driving Scenarios while Preserving Similarity to Expert Trajectoriescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Driving
  2. 2024Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving
  3. 2025LLaViDA: A Large Language Vision Driving Assistant for Explicit Reasoning and Enhanced Trajectory Planning
  4. 2026MAGNIFIED: RL Fine-tuning of Multimodal Large Language Models for Motion Planning
  5. 2026DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving
  6. 2025LAP: Fast LAtent Diffusion Planner for Autonomous Driving
  7. 2025AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Framework
  8. 2023GPT-Driver: Learning to Drive with GPT