# End-to-end Interpretable Neural Motion Planner

arXiv: https://arxiv.org/abs/2101.06679 · HTML: https://mlsanity.com/paper/2101.06679/

## Key facts

- Cited by 242 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 17 Jan 2021.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 18 of 18)

- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Baidu Apollo EM Motion Planner](https://mlsanity.com/paper/1807.08048/) (2018)
- [Driving Policy Transfer via Modularity and Abstraction](https://mlsanity.com/paper/1804.09364/) (2018)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [Conditional Affordance Learning for Driving in Urban Environments](https://mlsanity.com/paper/1806.06498/) (2018)
- [Search-based optimal motion planning for automated driving](https://mlsanity.com/paper/1803.04868/) (2018)

## Built on by (most cited first, 30 of 242)

- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024)
- [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving](https://mlsanity.com/paper/2311.16038/) (2023)
- [GenAD: Generative End-to-End Autonomous Driving](https://mlsanity.com/paper/2402.11502/) (2024)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [End-to-End Urban Driving by Imitating a Reinforcement Learning Coach](https://mlsanity.com/paper/2108.08265/) (2021)
- [Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes](https://mlsanity.com/paper/2305.10430/) (2023)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving](https://mlsanity.com/paper/2305.06242/) (2023)
- [Enhancing End-to-End Autonomous Driving with Latent World Model](https://mlsanity.com/paper/2406.08481/) (2024)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [A Language Agent for Autonomous Driving](https://mlsanity.com/paper/2311.10813/) (2023)
- [Behavior Generation with Latent Actions](https://mlsanity.com/paper/2403.03181/) (2024)
- [Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving](https://mlsanity.com/paper/2410.22313/) (2024)
- [Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization](https://mlsanity.com/paper/2104.05833/) (2021)
- [GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving](https://mlsanity.com/paper/2303.05760/) (2023)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)

## Abstract (arXiv)

In this paper, we propose a neural motion planner (NMP) for learning to drive autonomously in complex urban scenarios that include traffic-light handling, yielding, and interactions with multiple road-users. Towards this goal, we design a holistic model that takes as input raw LIDAR data and a HD map and produces interpretable intermediate representations in the form of 3D detections and their future trajectories, as well as a cost volume defining the goodness of each position that the self-driving car can take within the planning horizon. We then sample a set of diverse physically possible trajectories and choose the one with the minimum learned cost. Importantly, our cost volume is able to naturally capture multi-modality. We demonstrate the effectiveness of our approach in real-world driving data captured in several cities in North America. Our experiments show that the learned cost volume can generate safer planning than all the baselines.