# Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehicles

arXiv: https://arxiv.org/abs/1910.04586 · HTML: https://mlsanity.com/paper/1910.04586/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2019.

## Built on by (most cited first, 30 of 51)

- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [Diffusion-based Generation, Optimization, and Planning in 3D Scenes](https://mlsanity.com/paper/2301.06015/) (2023)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [Differentiable Raycasting for Self-supervised Occupancy Forecasting](https://mlsanity.com/paper/2210.01917/) (2022)
- [Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving](https://mlsanity.com/paper/2207.10422/) (2022)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving](https://mlsanity.com/paper/2212.08787/) (2022)
- [Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving](https://mlsanity.com/paper/2308.01471/) (2023)
- [PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation](https://mlsanity.com/paper/2109.11094/) (2021)
- [Rethinking Closed-loop Training for Autonomous Driving](https://mlsanity.com/paper/2306.15713/) (2023)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Learning Interpretable End-to-End Vision-Based Motion Planning for Autonomous Driving with Optical Flow Distillation](https://mlsanity.com/paper/2104.12861/) (2021)
- [Deep Multi-Task Learning for Joint Localization, Perception, and Prediction](https://mlsanity.com/paper/2101.06720/) (2021)
- [Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterization](https://mlsanity.com/paper/2103.01882/) (2021)
- [VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion](https://mlsanity.com/paper/2502.18042/) (2025)
- [Deep Structured Reactive Planning](https://mlsanity.com/paper/2101.06832/) (2021)
- [A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving](https://mlsanity.com/paper/2405.01440/) (2024)
- [End-to-End Interactive Prediction and Planning with Optical Flow Distillation for Autonomous Driving](https://mlsanity.com/paper/2104.08862/) (2021)
- [DeTra: A Unified Model for Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2406.04426/) (2024)
- [The Reasonable Crowd: Towards evidence-based and interpretable models of driving behavior](https://mlsanity.com/paper/2107.13507/) (2021)
- [Hybrid Reasoning Based on Large Language Models for Autonomous Car Driving](https://mlsanity.com/paper/2402.13602/) (2024)
- [QuAD: Query-based Interpretable Neural Motion Planning for Autonomous Driving](https://mlsanity.com/paper/2404.01486/) (2024)
- [Learning to Drive via Asymmetric Self-Play](https://mlsanity.com/paper/2409.18218/) (2024)

## Abstract (arXiv)

The motion planners used in self-driving vehicles need to generate trajectories that are safe, comfortable, and obey the traffic rules. This is usually achieved by two modules: behavior planner, which handles high-level decisions and produces a coarse trajectory, and trajectory planner that generates a smooth, feasible trajectory for the duration of the planning horizon. These planners, however, are typically developed separately, and changes in the behavior planner might affect the trajectory planner in unexpected ways. Furthermore, the final trajectory outputted by the trajectory planner might differ significantly from the one generated by the behavior planner, as they do not share the same objective. In this paper, we propose a jointly learnable behavior and trajectory planner. Unlike most existing learnable motion planners that address either only behavior planning, or use an uninterpretable neural network to represent the entire logic from sensors to driving commands, our approach features an interpretable cost function on top of perception, prediction and vehicle dynamics, and a joint learning algorithm that learns a shared cost function employed by our behavior and trajectory components. Experiments on real-world self-driving data demonstrate that jointly learned planner performs significantly better in terms of both similarity to human driving and other safety metrics, compared to baselines that do not adopt joint behavior and trajectory learning.