# Safe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Experts

arXiv: https://arxiv.org/abs/2211.02131 · HTML: https://mlsanity.com/paper/2211.02131/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations](https://mlsanity.com/paper/2008.05930/) (2020)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [SimNet: Learning Reactive Self-driving Simulations from Real-world Observations](https://mlsanity.com/paper/2105.12332/) (2021)
- [SafetyNet: Safe planning for real-world self-driving vehicles using machine-learned policies](https://mlsanity.com/paper/2109.13602/) (2021)
- [Identifying Driver Interactions via Conditional Behavior Prediction](https://mlsanity.com/paper/2104.09959/) (2021)
- [Variational End-to-End Navigation and Localization](https://mlsanity.com/paper/1811.10119/) (2018)
- [PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation](https://mlsanity.com/paper/2109.11094/) (2021)
- [Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty](https://mlsanity.com/paper/2104.12446/) (2021)

## Built on by (most cited first, 29 of 29)

- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [GenAD: Generative End-to-End Autonomous Driving](https://mlsanity.com/paper/2402.11502/) (2024)
- [PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Driving](https://mlsanity.com/paper/2404.14327/) (2024)
- [HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decoder](https://mlsanity.com/paper/2503.08612/) (2025)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Rethinking Imitation-based Planner for Autonomous Driving](https://mlsanity.com/paper/2309.10443/) (2023)
- [Plan-R1: Safe and Feasible Trajectory Planning as Language Modeling](https://mlsanity.com/paper/2505.17659/) (2025)
- [Hybrid-Prediction Integrated Planning for Autonomous Driving](https://mlsanity.com/paper/2402.02426/) (2024)
- [AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Dataset](https://mlsanity.com/paper/2404.02429/) (2024)
- [Enhancing Autonomous Driving Safety with Collision Scenario Integration](https://mlsanity.com/paper/2503.03957/) (2025)
- [CRITERIA: a New Benchmarking Paradigm for Evaluating Trajectory Prediction Models for Autonomous Driving](https://mlsanity.com/paper/2310.07794/) (2023)
- [Learning Multiple Probabilistic Decisions from Latent World Model in Autonomous Driving](https://mlsanity.com/paper/2409.15730/) (2024)
- [Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Survey](https://mlsanity.com/paper/2404.16356/) (2024)
- [GEMINUS: Dual-aware Global and Scene-Adaptive Mixture-of-Experts for End-to-End Autonomous Driving](https://mlsanity.com/paper/2507.14456/) (2025)
- [Pioneering SE(2)-Equivariant Trajectory Planning for Automated Driving](https://mlsanity.com/paper/2403.11304/) (2024)
- [Perfect Prediction or Plenty of Proposals? What Matters Most in Planning for Autonomous Driving](https://mlsanity.com/paper/2510.15505/) (2025)
- [Imagination-Augmented Hierarchical Reinforcement Learning for Safe and Interactive Autonomous Driving in Urban Environments](https://mlsanity.com/paper/2311.10309/) (2023)
- [PP-TIL: Personalized Planning for Autonomous Driving with Instance-based Transfer Imitation Learning](https://mlsanity.com/paper/2407.18569/) (2024)
- [ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Experts](https://mlsanity.com/paper/2511.11740/) (2025)
- [CoCoGesture: Toward Coherent Co-speech 3D Gesture Generation in the Wild](https://mlsanity.com/paper/2405.16874/) (2024)
- [The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications](https://mlsanity.com/paper/2602.08019/) (2026)
- [Trajectory Planning for Autonomous Vehicle Using Iterative Reward Prediction in Reinforcement Learning](https://mlsanity.com/paper/2404.12079/) (2024)
- [Utilizing Navigation Paths to Generate Target Points for Enhanced End-to-End Autonomous Driving Planning](https://mlsanity.com/paper/2406.08349/) (2024)
- [Hybrid Imitation-Learning Motion Planner for Urban Driving](https://mlsanity.com/paper/2409.02871/) (2024)
- [CAFE-AD: Cross-Scenario Adaptive Feature Enhancement for Trajectory Planning in Autonomous Driving](https://mlsanity.com/paper/2504.06584/) (2025)
- [PIE: Perception and Interaction Enhanced End-to-End Motion Planning for Autonomous Driving](https://mlsanity.com/paper/2509.18609/) (2025)
- [Driving is a Game: Combining Planning and Prediction with Bayesian Iterative Best Response](https://mlsanity.com/paper/2512.03936/) (2025)
- [STAGE: STyle-controllable Action GEneration for personalized autonomous driving](https://mlsanity.com/paper/2607.29517/) (2026)
- [Risk-Aware World Modeling with Flow-Guided Occupancy Evolution for Selective Trajectory Planning in Automated Driving](https://mlsanity.com/paper/2609.18442/) (2026)

## Abstract (arXiv)

The goal of autonomous vehicles is to navigate public roads safely and comfortably. To enforce safety, traditional planning approaches rely on handcrafted rules to generate trajectories. Machine learning-based systems, on the other hand, scale with data and are able to learn more complex behaviors. However, they often ignore that agents and self-driving vehicle trajectory distributions can be leveraged to improve safety. In this paper, we propose modeling a distribution over multiple future trajectories for both the self-driving vehicle and other road agents, using a unified neural network architecture for prediction and planning. During inference, we select the planning trajectory that minimizes a cost taking into account safety and the predicted probabilities. Our approach does not depend on any rule-based planners for trajectory generation or optimization, improves with more training data and is simple to implement. We extensively evaluate our method through a realistic simulator and show that the predicted trajectory distribution corresponds to different driving profiles. We also successfully deploy it on a self-driving vehicle on urban public roads, confirming that it drives safely without compromising comfort. The code for training and testing our model on a public prediction dataset and the video of the road test are available at https://woven.mobi/safepathnet