# Learning to Predict Vehicle Trajectories with Model-based Planning

arXiv: https://arxiv.org/abs/2103.04027 · HTML: https://mlsanity.com/paper/2103.04027/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 6 Mar 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving](https://mlsanity.com/paper/1808.05819/) (2018)
- [Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting](https://mlsanity.com/paper/1910.03650/) (2019)
- [PiP: Planning-informed Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/2003.11476/) (2020)
- [Pedestrian Prediction by Planning using Deep Neural Networks](https://mlsanity.com/paper/1706.05904/) (2017)

## Built on by (most cited first, 30 of 63)

- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2109.01827/) (2021)
- [HOME: Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2105.10968/) (2021)
- [ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries](https://mlsanity.com/paper/2208.01582/) (2022)
- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling](https://mlsanity.com/paper/2110.06607/) (2021)
- [M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction](https://mlsanity.com/paper/2202.11884/) (2022)
- [Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders](https://mlsanity.com/paper/2308.09882/) (2023)
- [ProphNet: Efficient Agent-Centric Motion Forecasting with Anchor-Informed Proposals](https://mlsanity.com/paper/2303.12071/) (2023)
- [FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphs](https://mlsanity.com/paper/2211.16197/) (2022)
- [ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptation](https://mlsanity.com/paper/2307.14187/) (2023)
- [Vehicle trajectory prediction works, but not everywhere](https://mlsanity.com/paper/2112.03909/) (2021)
- [PPAD: Iterative Interactions of Prediction and Planning for End-to-end Autonomous Driving](https://mlsanity.com/paper/2311.08100/) (2023)
- [SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous Driving](https://mlsanity.com/paper/2206.14116/) (2022)
- [MacFormer: Map-Agent Coupled Transformer for Real-time and Robust Trajectory Prediction](https://mlsanity.com/paper/2308.10280/) (2023)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [DriveGPT: Scaling Autoregressive Behavior Models for Driving](https://mlsanity.com/paper/2412.14415/) (2024)
- [Dynamic Scenario Representation Learning for Motion Forecasting with Heterogeneous Graph Convolutional Recurrent Networks](https://mlsanity.com/paper/2303.04364/) (2023)
- [R-Pred: Two-Stage Motion Prediction Via Tube-Query Attention-Based Trajectory Refinement](https://mlsanity.com/paper/2211.08609/) (2022)
- [TrajGen: Generating Realistic and Diverse Trajectories with Reactive and Feasible Agent Behaviors for Autonomous Driving](https://mlsanity.com/paper/2203.16792/) (2022)
- [Bootstrap Motion Forecasting With Self-Consistent Constraints](https://mlsanity.com/paper/2204.05859/) (2022)
- [Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions](https://mlsanity.com/paper/2307.07527/) (2023)
- [Trajectory Prediction with Graph-based Dual-scale Context Fusion](https://mlsanity.com/paper/2111.01592/) (2021)
- [Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning](https://mlsanity.com/paper/2509.23993/) (2025)
- [MPA: MultiPath++ Based Architecture for Motion Prediction](https://mlsanity.com/paper/2206.10041/) (2022)
- [Collaborative Uncertainty Benefits Multi-Agent Multi-Modal Trajectory Forecasting](https://mlsanity.com/paper/2207.05195/) (2022)
- [HYPER: Learned Hybrid Trajectory Prediction via Factored Inference and Adaptive Sampling](https://mlsanity.com/paper/2110.02344/) (2021)
- [Diverse Multiple Trajectory Prediction Using a Two-stage Prediction Network Trained with Lane Loss](https://mlsanity.com/paper/2206.08641/) (2022)
- [MFTraj: Map-Free, Behavior-Driven Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/2405.01266/) (2024)
- [KEMP: Keyframe-Based Hierarchical End-to-End Deep Model for Long-Term Trajectory Prediction](https://mlsanity.com/paper/2205.04624/) (2022)

## Abstract (arXiv)

Predicting the future trajectories of on-road vehicles is critical for autonomous driving. In this paper, we introduce a novel prediction framework called PRIME, which stands for Prediction with Model-based Planning. Unlike recent prediction works that utilize neural networks to model scene context and produce unconstrained trajectories, PRIME is designed to generate accurate and feasibility-guaranteed future trajectory predictions. PRIME guarantees the trajectory feasibility by exploiting a model-based generator to produce future trajectories under explicit constraints and enables accurate multimodal prediction by utilizing a learning-based evaluator to select future trajectories. We conduct experiments on the large-scale Argoverse Motion Forecasting Benchmark, where PRIME outperforms the state-of-the-art methods in prediction accuracy, feasibility, and robustness under imperfect tracking.