# TIP: Task-Informed Motion Prediction for Intelligent Vehicles

arXiv: https://arxiv.org/abs/2110.08750 · HTML: https://mlsanity.com/paper/2110.08750/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 50 references that are in the map.
- First posted on arXiv 17 Oct 2021.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting](https://mlsanity.com/paper/1910.03650/) (2019)
- [PiP: Planning-informed Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/2003.11476/) (2020)
- [Neural Approximate Sufficient Statistics for Implicit Models](https://mlsanity.com/paper/2010.10079/) (2020)
- [Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data](https://mlsanity.com/paper/2001.03093/) (2020)
- [Probabilistic Vehicle Trajectory Prediction over Occupancy Grid Map via Recurrent Neural Network](https://mlsanity.com/paper/1704.07049/) (2017)
- [Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehicles](https://mlsanity.com/paper/1910.04586/) (2019)
- [Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction](https://mlsanity.com/paper/1710.09483/) (2017)
- [Identifying Driver Interactions via Conditional Behavior Prediction](https://mlsanity.com/paper/2104.09959/) (2021)
- [DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Sampling](https://mlsanity.com/paper/1911.12736/) (2019)
- [Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Network](https://mlsanity.com/paper/1510.02175/) (2015)
- [Deep Learning-based Vehicle Behaviour Prediction For Autonomous Driving Applications: A Review](https://mlsanity.com/paper/1912.11676/) (2019)
- [BARK: Open Behavior Benchmarking in Multi-Agent Environments](https://mlsanity.com/paper/2003.02604/) (2020)
- [Risk-Sensitive Sequential Action Control with Multi-Modal Human Trajectory Forecasting for Safe Crowd-Robot Interaction](https://mlsanity.com/paper/2009.05702/) (2020)
- [Injecting Planning-Awareness into Prediction and Detection Evaluation](https://mlsanity.com/paper/2110.03270/) (2021)

## Built on by (most cited first, 8 of 8)

- [What Truly Matters in Trajectory Prediction for Autonomous Driving?](https://mlsanity.com/paper/2306.15136/) (2023)
- [Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery](https://mlsanity.com/paper/2402.01974/) (2024)
- [MS-Net: A Multi-Path Sparse Model for Motion Prediction in Multi-Scenes](https://mlsanity.com/paper/2403.00353/) (2024)
- [P4P: Conflict-Aware Motion Prediction for Planning in Autonomous Driving](https://mlsanity.com/paper/2211.01634/) (2022)
- [NashFormer: Leveraging Local Nash Equilibria for Semantically Diverse Trajectory Prediction](https://mlsanity.com/paper/2305.17600/) (2023)
- [Autonomous vehicle decision and control through reinforcement learning with traffic flow randomization](https://mlsanity.com/paper/2403.02882/) (2024)
- [Controllable Generative Trajectory Prediction via Weak Preference Alignment](https://mlsanity.com/paper/2510.10731/) (2025)
- [Chatting about Conditional Trajectory Prediction](https://mlsanity.com/paper/2604.18126/) (2026)

## Abstract (arXiv)

When predicting trajectories of road agents, motion predictors usually approximate the future distribution by a limited number of samples. This constraint requires the predictors to generate samples that best support the task given task specifications. However, existing predictors are often optimized and evaluated via task-agnostic measures without accounting for the use of predictions in downstream tasks, and thus could result in sub-optimal task performance. In this paper, we propose a task-informed motion prediction model that better supports the tasks through its predictions, by jointly reasoning about prediction accuracy and the utility of the downstream tasks, which is commonly used to evaluate the task performance. The task utility function does not require the full task information, but rather a specification of the utility of the task, resulting in predictors that serve a wide range of downstream tasks. We demonstrate our approach on two use cases of common decision making tasks and their utility functions, in the context of autonomous driving and parallel autonomy. Experiment results show that our predictor produces accurate predictions that improve the task performance by a large margin in both tasks when compared to task-agnostic baselines on the Waymo Open Motion dataset.