# The Importance of Prior Knowledge in Precise Multimodal Prediction

arXiv: https://arxiv.org/abs/2006.02636 · HTML: https://mlsanity.com/paper/2006.02636/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2020.

## Built on by (most cited first, 25 of 25)

- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [GoRela: Go Relative for Viewpoint-Invariant Motion Forecasting](https://mlsanity.com/paper/2211.02545/) (2022)
- [ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning](https://mlsanity.com/paper/2206.13387/) (2022)
- [Vehicle trajectory prediction works, but not everywhere](https://mlsanity.com/paper/2112.03909/) (2021)
- [Divide-and-Conquer for Lane-Aware Diverse Trajectory Prediction](https://mlsanity.com/paper/2104.08277/) (2021)
- [DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles](https://mlsanity.com/paper/2212.06437/) (2022)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [Deep Multi-Task Learning for Joint Localization, Perception, and Prediction](https://mlsanity.com/paper/2101.06720/) (2021)
- [Propagating State Uncertainty Through Trajectory Forecasting](https://mlsanity.com/paper/2110.03267/) (2021)
- [Hierarchical Latent Structure for Multi-Modal Vehicle Trajectory Forecasting](https://mlsanity.com/paper/2207.04624/) (2022)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Diverse Multiple Trajectory Prediction Using a Two-stage Prediction Network Trained with Lane Loss](https://mlsanity.com/paper/2206.08641/) (2022)
- [Context-Aware Timewise VAEs for Real-Time Vehicle Trajectory Prediction](https://mlsanity.com/paper/2302.10873/) (2023)
- [Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2203.04421/) (2022)
- [Diverse Controllable Diffusion Policy with Signal Temporal Logic](https://mlsanity.com/paper/2503.02924/) (2025)
- [Joint Perception and Prediction for Autonomous Driving: A Survey](https://mlsanity.com/paper/2412.14088/) (2024)
- [Safety-Oriented Pedestrian Motion and Scene Occupancy Forecasting](https://mlsanity.com/paper/2101.02385/) (2021)
- [Uncertainty-Aware Trajectory Prediction via Rule-Regularized Heteroscedastic Deep Classification](https://mlsanity.com/paper/2504.13111/) (2025)
- [Efficient Data Representation for Motion Forecasting: A Scene-Specific Trajectory Set Approach](https://mlsanity.com/paper/2407.20732/) (2024)
- [ProspectNet: Weighted Conditional Attention for Future Interaction Modeling in Behavior Prediction](https://mlsanity.com/paper/2208.13848/) (2022)
- [To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software](https://mlsanity.com/paper/2508.20892/) (2025)
- [Class-Incremental Motion Forecasting](https://mlsanity.com/paper/2603.09420/) (2026)
- [GAD-Generative Learning for HD Map-Free Autonomous Driving](https://mlsanity.com/paper/2405.00515/) (2024)
- [Robust Planning for Autonomous Driving via Mixed Adversarial Diffusion Predictions](https://mlsanity.com/paper/2505.12327/) (2025)

## Abstract (arXiv)

Roads have well defined geometries, topologies, and traffic rules. While this has been widely exploited in motion planning methods to produce maneuvers that obey the law, little work has been devoted to utilize these priors in perception and motion forecasting methods. In this paper we propose to incorporate these structured priors as a loss function. In contrast to imposing hard constraints, this approach allows the model to handle non-compliant maneuvers when those happen in the real world. Safe motion planning is the end goal, and thus a probabilistic characterization of the possible future developments of the scene is key to choose the plan with the lowest expected cost. Towards this goal, we design a framework that leverages REINFORCE to incorporate non-differentiable priors over sample trajectories from a probabilistic model, thus optimizing the whole distribution. We demonstrate the effectiveness of our approach on real-world self-driving datasets containing complex road topologies and multi-agent interactions. Our motion forecasts not only exhibit better precision and map understanding, but most importantly result in safer motion plans taken by our self-driving vehicle. We emphasize that despite the importance of this evaluation, it has been often overlooked by previous perception and motion forecasting works.