# Interactive Joint Planning for Autonomous Vehicles

arXiv: https://arxiv.org/abs/2310.18301 · HTML: https://mlsanity.com/paper/2310.18301/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving.
- Builds on 15 of its 45 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Placed in

- End-to-end autonomous driving / [Imitation-learning end-to-end planners](https://mlsanity.com/driving/imitation-learning-end-to-end-planners/)

## Builds on (most cited first, 15 of 15)

- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning](https://mlsanity.com/paper/2206.13387/) (2022)
- [MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control](https://mlsanity.com/paper/2009.07517/) (2020)
- [Game-Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems](https://mlsanity.com/paper/1608.08589/) (2016)
- [Tree-structured Policy Planning with Learned Behavior Models](https://mlsanity.com/paper/2301.11902/) (2023)
- [Interactive multi-modal motion planning with Branch Model Predictive Control](https://mlsanity.com/paper/2109.05128/) (2021)
- [Winding Through: Crowd Navigation via Topological Invariance](https://mlsanity.com/paper/2109.05084/) (2021)
- [Multimodal Trajectory Prediction via Topological Invariance for Navigation at Uncontrolled Intersections](https://mlsanity.com/paper/2011.03894/) (2020)
- [Path Homotopy Invariants and their Application to Optimal Trajectory Planning](https://mlsanity.com/paper/1710.02871/) (2017)

## Built on by (most cited first, 13 of 13)

- [Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning](https://mlsanity.com/paper/2410.05582/) (2024)
- [Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs](https://mlsanity.com/paper/2404.17833/) (2024)
- [Language-Driven Active Learning for Diverse Open-Set 3D Object Detection](https://mlsanity.com/paper/2404.12856/) (2024)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latent](https://mlsanity.com/paper/2311.18307/) (2023)
- [Predictive Planner for Autonomous Driving with Consistency Models](https://mlsanity.com/paper/2502.08033/) (2025)
- [Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives](https://mlsanity.com/paper/2505.09074/) (2025)
- [SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modeling](https://mlsanity.com/paper/2506.21041/) (2025)
- [MTR-VP: Towards End-to-End Trajectory Planning through Context-Driven Image Encoding and Multiple Trajectory Prediction](https://mlsanity.com/paper/2511.22181/) (2025)
- [Mode Collapse Happens: Evaluating Critical Interactions in Joint Trajectory Prediction Models](https://mlsanity.com/paper/2506.23164/) (2025)
- [Safe and Nonconservative Contingency Planning for Autonomous Vehicles via Online Learning-Based Reachable Set Barriers](https://mlsanity.com/paper/2509.07464/) (2025)
- [Flow-Opt: Scalable Centralized Multi-Robot Trajectory Optimization with Flow Matching and Differentiable Optimization](https://mlsanity.com/paper/2510.09204/) (2025)
- [Agent-driven Long-tail Simulation for Autonomous Driving](https://mlsanity.com/paper/2607.04331/) (2026)

## Abstract (arXiv)

In highly interactive driving scenarios, the actions of one agent greatly influences those of its neighbors. Planning safe motions for autonomous vehicles in such interactive environments, therefore, requires reasoning about the impact of the ego's intended motion plan on nearby agents' behavior. Deep-learning-based models have recently achieved great success in trajectory prediction and many models in the literature allow for ego-conditioned prediction. However, leveraging ego-conditioned prediction remains challenging in downstream planning due to the complex nature of neural networks, limiting the planner structure to simple ones, e.g., sampling-based planner. Despite their ability to generate fine-grained high-quality motion plans, it is difficult for gradient-based planning algorithms, such as model predictive control (MPC), to leverage ego-conditioned prediction due to their iterative nature and need for gradient. We present Interactive Joint Planning (IJP) that bridges MPC with learned prediction models in a computationally scalable manner to provide us the best of both the worlds. In particular, IJP jointly optimizes over the behavior of the ego and the surrounding agents and leverages deep-learned prediction models as prediction priors that the join trajectory optimization tries to stay close to. Furthermore, by leveraging homotopy classes, our joint optimizer searches over diverse motion plans to avoid getting stuck at local minima. Closed-loop simulation result shows that IJP significantly outperforms the baselines that are either without joint optimization or running sampling-based planning.