# Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Driving

arXiv: https://arxiv.org/abs/2607.10438 · HTML: https://mlsanity.com/paper/2607.10438/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 34 of its 54 references that are in the map.
- First posted on arXiv 11 Jul 2026.

## Placed in

- End-to-end autonomous driving / [Large language models for driving](https://mlsanity.com/driving/large-language-models-for-driving/)

## Builds on (most cited first, 30 of 34)

- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving](https://mlsanity.com/paper/2303.05760/) (2023)
- [Diffusion-Based Planning for Autonomous Driving with Flexible Guidance](https://mlsanity.com/paper/2501.15564/) (2025)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [Reason2Drive: Towards Interpretable and Chain-based Reasoning for Autonomous Driving](https://mlsanity.com/paper/2312.03661/) (2023)
- [Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients](https://mlsanity.com/paper/2109.13333/) (2021)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities](https://mlsanity.com/paper/2309.16739/) (2023)
- [PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Driving](https://mlsanity.com/paper/2404.14327/) (2024)
- [Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving](https://mlsanity.com/paper/2109.06446/) (2021)
- [Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving](https://mlsanity.com/paper/2207.10422/) (2022)
- [Theseus: A Library for Differentiable Nonlinear Optimization](https://mlsanity.com/paper/2207.09442/) (2022)
- [SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models](https://mlsanity.com/paper/2407.00952/) (2024)
- [Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2404.06448/) (2024)
- [From Prediction to Planning With Goal Conditioned Lane Graph Traversals](https://mlsanity.com/paper/2302.07753/) (2023)
- [Asynchronous Large Language Model Enhanced Planner for Autonomous Driving](https://mlsanity.com/paper/2406.14556/) (2024)
- [Hierarchical Split Federated Learning: Convergence Analysis and System Optimization](https://mlsanity.com/paper/2412.07197/) (2024)
- [Optimal Resource Allocation for U-Shaped Parallel Split Learning](https://mlsanity.com/paper/2308.08896/) (2023)
- [HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models](https://mlsanity.com/paper/2505.02795/) (2025)
- [IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengers](https://mlsanity.com/paper/2410.02592/) (2024)
- [Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula](https://mlsanity.com/paper/2212.01375/) (2022)
- [LeGo-Drive: Language-enhanced Goal-oriented Closed-Loop End-to-End Autonomous Driving](https://mlsanity.com/paper/2403.20116/) (2024)
- [HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts](https://mlsanity.com/paper/2601.00583/) (2026)
- [LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends](https://mlsanity.com/paper/2410.15281/) (2024)

## Abstract (arXiv)

Autonomous driving planning is a key component of IoT-enabled intelligent transportation systems, requiring vehicles to generate safe, efficient, and executable trajectories in complex urban environments from multi-source contextual information. While imitation learning (IL) has shown promise on large-scale datasets, IL-based planners still suffer from limited coverage of complex long-tail interactions, weak consistency with downstream constrained refinement, and insufficient use of high level scene semantics under real time constraints. To address these issues, this paper proposes a large language model (LLM) enhanced differentiable trajectory planning framework for IoT-enabled autonomous driving. Specifically, we introduce a surrounding agent centric data augmentation strategy to reorganize sur rounding agent trajectories as additional planning supervision, thereby improving the training distribution without collecting additional raw data. We further design a complexity-aware asyn chronous LLM-based semantic enhancement module to extract scene-related high-level semantic features with controlled online overhead. In addition, a differentiable optimization module is incorporated to refine generated trajectories with explicit residual penalties while backpropagating optimization gradients to the upstream planner. Experiments show that the proposed method achieves the best overall scores of 83.63 and 78.29 on the nuPlan closed-loop nonreactive and reactive Hard20 benchmarks, respectively, and CARLA-ROS tests further verify its online deployment and real time closed-loop execution capability.