# Injecting Knowledge in Data-driven Vehicle Trajectory Predictors

arXiv: https://arxiv.org/abs/2103.04854 · HTML: https://mlsanity.com/paper/2103.04854/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 66 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](https://mlsanity.com/paper/1903.12394/) (2019)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning](https://mlsanity.com/paper/1905.06088/) (2019)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling](https://mlsanity.com/paper/1710.11431/) (2017)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Constrained Convolutional Neural Networks for Weakly Supervised Segmentation](https://mlsanity.com/paper/1506.03648/) (2015)
- [Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses](https://mlsanity.com/paper/1612.00197/) (2016)
- [Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction](https://mlsanity.com/paper/1906.03631/) (2019)
- [What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction](https://mlsanity.com/paper/1903.07933/) (2019)
- [Conditional Generative Neural System for Probabilistic Trajectory Prediction](https://mlsanity.com/paper/1905.01631/) (2019)
- [Social NCE: Contrastive Learning of Socially-aware Motion Representations](https://mlsanity.com/paper/2012.11717/) (2020)
- [CAR-Net: Clairvoyant Attentive Recurrent Network](https://mlsanity.com/paper/1711.10061/) (2017)
- [DR-RNN: A deep residual recurrent neural network for model reduction](https://mlsanity.com/paper/1709.00939/) (2017)
- [Towards Full Automated Drive in Urban Environments: A Demonstration in GoMentum Station, California](https://mlsanity.com/paper/1705.01187/) (2017)
- [Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey](https://mlsanity.com/paper/2005.10691/) (2020)
- [Cooperation-Aware Lane Change Maneuver in Dense Traffic based on Model Predictive Control with Recurrent Neural Network](https://mlsanity.com/paper/1909.05665/) (2019)
- [Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network](https://mlsanity.com/paper/2002.06241/) (2020)
- [Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions](https://mlsanity.com/paper/1908.00219/) (2019)
- [A Learning-based Stochastic MPC Design for Cooperative Adaptive Cruise Control to Handle Interfering Vehicles](https://mlsanity.com/paper/1802.09356/) (2018)

## Built on by (most cited first, 18 of 18)

- [UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction](https://mlsanity.com/paper/2403.15098/) (2024)
- [Vehicle trajectory prediction works, but not everywhere](https://mlsanity.com/paper/2112.03909/) (2021)
- [Are socially-aware trajectory prediction models really socially-aware?](https://mlsanity.com/paper/2108.10879/) (2021)
- [Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-based Approaches](https://mlsanity.com/paper/2111.06740/) (2021)
- [Towards Knowledge-driven Autonomous Driving](https://mlsanity.com/paper/2312.04316/) (2023)
- [I-24 MOTION: An instrument for freeway traffic science](https://mlsanity.com/paper/2301.11198/) (2023)
- [Safety-aware Motion Prediction with Unseen Vehicles for Autonomous Driving](https://mlsanity.com/paper/2109.01510/) (2021)
- [Towards Explainable Motion Prediction using Heterogeneous Graph Representations](https://mlsanity.com/paper/2212.03806/) (2022)
- [Social-Pose: Enhancing Trajectory Prediction with Human Body Pose](https://mlsanity.com/paper/2507.22742/) (2025)
- [Hierarchical Motion Encoder-Decoder Network for Trajectory Forecasting](https://mlsanity.com/paper/2111.13324/) (2021)
- [Hierarchical Light Transformer Ensembles for Multimodal Trajectory Forecasting](https://mlsanity.com/paper/2403.17678/) (2024)
- [LoRD: Adapting Differentiable Driving Policies to Distribution Shifts](https://mlsanity.com/paper/2410.09681/) (2024)
- [Uncertainty-Aware Trajectory Prediction via Rule-Regularized Heteroscedastic Deep Classification](https://mlsanity.com/paper/2504.13111/) (2025)
- [Informed Priors for Knowledge Integration in Trajectory Prediction](https://mlsanity.com/paper/2211.00348/) (2022)
- [Informed Spectral Normalized Gaussian Processes for Trajectory Prediction](https://mlsanity.com/paper/2403.11966/) (2024)
- [FeaXDrive: Feasibility-aware Trajectory-Centric Diffusion Planning for End-to-End Autonomous Driving](https://mlsanity.com/paper/2604.12656/) (2026)
- [Knowledge Integration Strategies in Autonomous Vehicle Prediction and Planning: A Comprehensive Survey](https://mlsanity.com/paper/2502.10477/) (2025)
- [Markovian Dynamics Enforcer: Feasibility Preserving Correction on Learned Dynamics Manifolds](https://mlsanity.com/paper/2609.39888/) (2026)

## Abstract (arXiv)

Vehicle trajectory prediction tasks have been commonly tackled from two distinct perspectives: either with knowledge-driven methods or more recently with data-driven ones. On the one hand, we can explicitly implement domain-knowledge or physical priors such as anticipating that vehicles will follow the middle of the roads. While this perspective leads to feasible outputs, it has limited performance due to the difficulty to hand-craft complex interactions in urban environments. On the other hand, recent works use data-driven approaches which can learn complex interactions from the data leading to superior performance. However, generalization, textiti.e., having accurate predictions on unseen data, is an issue leading to unrealistic outputs. In this paper, we propose to learn a "Realistic Residual Block" (RRB), which effectively connects these two perspectives. Our RRB takes any off-the-shelf knowledge-driven model and finds the required residuals to add to the knowledge-aware trajectory. Our proposed method outputs realistic predictions by confining the residual range and taking into account its uncertainty. We also constrain our output with Model Predictive Control (MPC) to satisfy kinematic constraints. Using a publicly available dataset, we show that our method outperforms previous works in terms of accuracy and generalization to new scenes. We will release our code and data split here: https://github.com/vita-epfl/RRB.