# HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attention

arXiv: https://arxiv.org/abs/2404.06351 · HTML: https://mlsanity.com/paper/2404.06351/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [MotionLM: Multi-Agent Motion Forecasting as Language Modeling](https://mlsanity.com/paper/2309.16534/) (2023)
- [GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2109.01827/) (2021)
- [Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction](https://mlsanity.com/paper/2104.00563/) (2021)
- [HOME: Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2105.10968/) (2021)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals](https://mlsanity.com/paper/2106.15004/) (2021)
- [HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding](https://mlsanity.com/paper/2205.09753/) (2022)
- [THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling](https://mlsanity.com/paper/2110.06607/) (2021)
- [TPCN: Temporal Point Cloud Networks for Motion Forecasting](https://mlsanity.com/paper/2103.03067/) (2021)
- [Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles](https://mlsanity.com/paper/1606.07839/) (2016)
- [Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders](https://mlsanity.com/paper/2308.09882/) (2023)
- [LaPred: Lane-Aware Prediction of Multi-Modal Future Trajectories of Dynamic Agents](https://mlsanity.com/paper/2104.00249/) (2021)
- [GANet: Goal Area Network for Motion Forecasting](https://mlsanity.com/paper/2209.09723/) (2022)

## Built on by (most cited first, 30 of 38)

- [Plan-R1: Safe and Feasible Trajectory Planning as Language Modeling](https://mlsanity.com/paper/2505.17659/) (2025)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Trajectory Prediction Meets Large Language Models: A Survey](https://mlsanity.com/paper/2506.03408/) (2025)
- [ComDrive: Comfort-Oriented End-to-End Autonomous Driving](https://mlsanity.com/paper/2410.05051/) (2024)
- [Motion Forecasting in Continuous Driving](https://mlsanity.com/paper/2410.06007/) (2024)
- [Occupancy World Model for Robots](https://mlsanity.com/paper/2505.05512/) (2025)
- [TrajDiff: End-to-end Autonomous Driving without Perception Annotation](https://mlsanity.com/paper/2512.00723/) (2025)
- [Risk-Aware Vehicle Trajectory Prediction Under Safety-Critical Scenarios](https://mlsanity.com/paper/2407.13480/) (2024)
- [SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction](https://mlsanity.com/paper/2410.08669/) (2024)
- [DONUT: A Decoder-Only Model for Trajectory Prediction](https://mlsanity.com/paper/2506.06854/) (2025)
- [Foresight in Motion: Reinforcing Trajectory Prediction with Reward Heuristics](https://mlsanity.com/paper/2507.12083/) (2025)
- [Generative Active Learning for Long-tail Trajectory Prediction via Controllable Diffusion Model](https://mlsanity.com/paper/2507.22615/) (2025)
- [Building Real-time Awareness of Out-of-distribution in Trajectory Prediction for Autonomous Vehicles](https://mlsanity.com/paper/2409.17277/) (2024)
- [Den-TP: A Density-Balanced Data Curation and Evaluation Framework for Trajectory Prediction](https://mlsanity.com/paper/2409.17385/) (2024)
- [Towards Generalizable Trajectory Prediction Using Dual-Level Representation Learning And Adaptive Prompting](https://mlsanity.com/paper/2501.04815/) (2025)
- [Remember and Recall: Associative-Memory-based Trajectory Prediction](https://mlsanity.com/paper/2410.02201/) (2024)
- [Knowledge-Informed Multi-Agent Trajectory Prediction at Signalized Intersections for Infrastructure-to-Everything](https://mlsanity.com/paper/2501.13461/) (2025)
- [GAMDTP: Dynamic Trajectory Prediction with Graph Attention Mamba Network](https://mlsanity.com/paper/2504.04862/) (2025)
- [DyTTP: Trajectory Prediction with Normalization-Free Transformers](https://mlsanity.com/paper/2504.05356/) (2025)
- [Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction](https://mlsanity.com/paper/2603.10597/) (2026)
- [Boundary-aware Decoupled Flow Networks for Realistic Extreme Rescaling](https://mlsanity.com/paper/2405.02941/) (2024)
- [Invertible Residual Rescaling Models](https://mlsanity.com/paper/2405.02945/) (2024)
- [LHPF: Look back the History and Plan for the Future in Autonomous Driving](https://mlsanity.com/paper/2411.17253/) (2024)
- [Adaptive Output Steps: FlexiSteps Network for Dynamic Trajectory Prediction](https://mlsanity.com/paper/2508.17797/) (2025)
- [Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling](https://mlsanity.com/paper/2603.01864/) (2026)
- [Rethinking Foundation Model Collaboration: Enhancing Specialized Models through Proxy Task Reasoning](https://mlsanity.com/paper/2606.31157/) (2026)
- [ILNet: Trajectory Prediction with Inverse Learning Attention for Enhancing Intention Capture](https://mlsanity.com/paper/2507.06531/) (2025)
- [CoPAD : Multi-source Trajectory Fusion and Cooperative Trajectory Prediction with Anchor-oriented Decoder in V2X Scenarios](https://mlsanity.com/paper/2509.15984/) (2025)
- [A Trajectory Generator for High-Density Traffic and Diverse Agent-Interaction Scenarios](https://mlsanity.com/paper/2510.02627/) (2025)
- [V2X-RECT: An Efficient V2X Trajectory Prediction Framework via Redundant Interaction Filtering and Tracking Error Correction](https://mlsanity.com/paper/2511.17941/) (2025)

## Abstract (arXiv)

Predicting the trajectories of road agents is essential for autonomous driving systems. The recent mainstream methods follow a static paradigm, which predicts the future trajectory by using a fixed duration of historical frames. These methods make the predictions independently even at adjacent time steps, which leads to potential instability and temporal inconsistency. As successive time steps have largely overlapping historical frames, their forecasting should have intrinsic correlation, such as overlapping predicted trajectories should be consistent, or be different but share the same motion goal depending on the road situation. Motivated by this, in this work, we introduce HPNet, a novel dynamic trajectory forecasting method. Aiming for stable and accurate trajectory forecasting, our method leverages not only historical frames including maps and agent states, but also historical predictions. Specifically, we newly design a Historical Prediction Attention module to automatically encode the dynamic relationship between successive predictions. Besides, it also extends the attention range beyond the currently visible window benefitting from the use of historical predictions. The proposed Historical Prediction Attention together with the Agent Attention and Mode Attention is further formulated as the Triple Factorized Attention module, serving as the core design of HPNet.Experiments on the Argoverse and INTERACTION datasets show that HPNet achieves state-of-the-art performance, and generates accurate and stable future trajectories. Our code are available at https://github.com/XiaolongTang23/HPNet.