# Multi-scale Temporal Fusion Transformer for Incomplete Vehicle Trajectory Prediction

arXiv: https://arxiv.org/abs/2409.00904 · HTML: https://mlsanity.com/paper/2409.00904/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Handling Incomplete Heterogeneous Data using VAEs](https://mlsanity.com/paper/1807.03653/) (2018)
- [Missing Data Imputation using Optimal Transport](https://mlsanity.com/paper/2002.03860/) (2020)
- [PiP: Planning-informed Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/2003.11476/) (2020)
- [Sequence-to-Sequence Prediction of Vehicle Trajectory via LSTM Encoder-Decoder Architecture](https://mlsanity.com/paper/1802.06338/) (2018)
- [Missing Data Imputation with Adversarially-trained Graph Convolutional Networks](https://mlsanity.com/paper/1905.01907/) (2019)
- [LAformer: Trajectory Prediction for Autonomous Driving with Lane-Aware Scene Constraints](https://mlsanity.com/paper/2302.13933/) (2023)
- [Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction](https://mlsanity.com/paper/2303.16005/) (2023)
- [SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction](https://mlsanity.com/paper/2403.11492/) (2024)
- [SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous Driving](https://mlsanity.com/paper/2206.14116/) (2022)
- [Hierarchical Latent Structure for Multi-Modal Vehicle Trajectory Forecasting](https://mlsanity.com/paper/2207.04624/) (2022)
- [Uncertainty estimation of pedestrian future trajectory using Bayesian approximation](https://mlsanity.com/paper/2205.01887/) (2022)
- [Modeling Multi-Vehicle Interaction Scenarios Using Gaussian Random Field](https://mlsanity.com/paper/1906.10307/) (2019)
- [Learning Individual Models for Imputation (Technical Report)](https://mlsanity.com/paper/2004.03436/) (2020)
- [MSTF: Multiscale Transformer for Incomplete Trajectory Prediction](https://mlsanity.com/paper/2407.05671/) (2024)

## Built on by (most cited first, 3 of 3)

- [YOLO-TS: Real-Time Traffic Sign Detection with Enhanced Accuracy Using Optimized Receptive Fields and Anchor-Free Fusion](https://mlsanity.com/paper/2410.17144/) (2024)
- [Focus Through Motion: RGB-Event Collaborative Token Sparsification for Efficient Object Detection](https://mlsanity.com/paper/2509.03872/) (2025)
- [Wavelet Policy: Imitation Learning in the Scale Domain with World Prior Memory](https://mlsanity.com/paper/2504.04991/) (2025)

## Abstract (arXiv)

Motion prediction plays an essential role in autonomous driving systems, enabling autonomous vehicles to achieve more accurate local-path planning and driving decisions based on predictions of the surrounding vehicles. However, existing methods neglect the potential missing values caused by object occlusion, perception failures, etc., which inevitably degrades the trajectory prediction performance in real traffic scenarios. To address this limitation, we propose a novel end-to-end framework for incomplete vehicle trajectory prediction, named Multi-scale Temporal Fusion Transformer (MTFT), which consists of the Multi-scale Attention Head (MAH) and the Continuity Representation-guided Multi-scale Fusion (CRMF) module. Specifically, the MAH leverages the multi-head attention mechanism to parallelly capture multi-scale motion representation of trajectory from different temporal granularities, thus mitigating the adverse effect of missing values on prediction. Furthermore, the multi-scale motion representation is input into the CRMF module for multi-scale fusion to obtain the robust temporal feature of the vehicle. During the fusion process, the continuity representation of vehicle motion is first extracted across time steps to guide the fusion, ensuring that the resulting temporal feature incorporates both detailed information and the overall trend of vehicle motion, which facilitates the accurate decoding of future trajectory that is consistent with the vehicle's motion trend. We evaluate the proposed model on four datasets derived from highway and urban traffic scenarios. The experimental results demonstrate its superior performance in the incomplete vehicle trajectory prediction task compared with state-of-the-art models, e.g., a comprehensive performance improvement of more than 39% on the HighD dataset.