# Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction

arXiv: https://arxiv.org/abs/2303.16005 · HTML: https://mlsanity.com/paper/2303.16005/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 84 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation](https://mlsanity.com/paper/2107.03502/) (2021)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [GAIN: Missing Data Imputation using Generative Adversarial Nets](https://mlsanity.com/paper/1806.02920/) (2018)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [BRITS: Bidirectional Recurrent Imputation for Time Series](https://mlsanity.com/paper/1805.10572/) (2018)
- [Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019](https://mlsanity.com/paper/1911.13288/) (2019)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [GP-VAE: Deep Probabilistic Time Series Imputation](https://mlsanity.com/paper/1907.04155/) (2019)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2204.08770/) (2022)
- [SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction](https://mlsanity.com/paper/1903.02793/) (2019)
- [SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2104.01528/) (2021)
- [On Adversarial Robustness of Trajectory Prediction for Autonomous Vehicles](https://mlsanity.com/paper/2201.05057/) (2022)
- [MaskGAN: Better Text Generation via Filling in the______](https://mlsanity.com/paper/1801.07736/) (2018)
- [Estimating Missing Data in Temporal Data Streams Using Multi-directional Recurrent Neural Networks](https://mlsanity.com/paper/1711.08742/) (2017)
- [Remember Intentions: Retrospective-Memory-based Trajectory Prediction](https://mlsanity.com/paper/2203.11474/) (2022)

## Built on by (most cited first, 30 of 47)

- [VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision](https://mlsanity.com/paper/2412.14446/) (2024)
- [Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction](https://mlsanity.com/paper/2403.18447/) (2024)
- [Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning](https://mlsanity.com/paper/2404.05218/) (2024)
- [Adapting to Length Shift: FlexiLength Network for Trajectory Prediction](https://mlsanity.com/paper/2404.00742/) (2024)
- [T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory](https://mlsanity.com/paper/2403.10052/) (2024)
- [GinAR: An End-To-End Multivariate Time Series Forecasting Model Suitable for Variable Missing](https://mlsanity.com/paper/2405.11333/) (2024)
- [Merlin: Multi-View Representation Learning for Robust Multivariate Time Series Forecasting with Unfixed Missing Rates](https://mlsanity.com/paper/2506.12459/) (2025)
- [Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sports](https://mlsanity.com/paper/2405.17680/) (2024)
- [Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2503.18589/) (2025)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [OOSTraj: Out-of-Sight Trajectory Prediction With Vision-Positioning Denoising](https://mlsanity.com/paper/2404.02227/) (2024)
- [Unified Spatial-Temporal Edge-Enhanced Graph Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2502.02504/) (2025)
- [TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sports](https://mlsanity.com/paper/2410.17785/) (2024)
- [Joint Pedestrian Trajectory Prediction through Posterior Sampling](https://mlsanity.com/paper/2404.00237/) (2024)
- [Pedestrian Trajectory Prediction with Missing Data: Datasets, Imputation, and Benchmarking](https://mlsanity.com/paper/2411.00174/) (2024)
- [Continuous Locomotive Crowd Behavior Generation](https://mlsanity.com/paper/2504.04756/) (2025)
- [Semantic Flow: Learning Semantic Field of Dynamic Scenes from Monocular Videos](https://mlsanity.com/paper/2404.05163/) (2024)
- [Improving Autonomous Driving Safety with POP: A Framework for Accurate Partially Observed Trajectory Predictions](https://mlsanity.com/paper/2309.15685/) (2023)
- [JointDiff: Bridging Continuous and Discrete in Multi-Agent Trajectory Generation](https://mlsanity.com/paper/2509.22522/) (2025)
- [MagiNet: Mask-Aware Graph Imputation Network for Incomplete Traffic Data](https://mlsanity.com/paper/2406.03511/) (2024)
- [Trajectory Imputation in Multi-Agent Sports with Derivative-Accumulating Self-Ensemble](https://mlsanity.com/paper/2408.10878/) (2024)
- [Multi-scale Temporal Fusion Transformer for Incomplete Vehicle Trajectory Prediction](https://mlsanity.com/paper/2409.00904/) (2024)
- [Den-TP: A Density-Balanced Data Curation and Evaluation Framework for Trajectory Prediction](https://mlsanity.com/paper/2409.17385/) (2024)
- [RealTraj: Towards Real-World Pedestrian Trajectory Forecasting](https://mlsanity.com/paper/2411.17376/) (2024)
- [Incomplete Graph Learning: A Comprehensive Survey](https://mlsanity.com/paper/2502.12412/) (2025)
- [MH-GIN: Multi-scale Heterogeneous Graph-based Imputation Network for AIS Data (Extended Version)](https://mlsanity.com/paper/2507.20362/) (2025)
- [Heteroscedastic Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2605.10717/) (2026)
- [Towards Predicting Any Human Trajectory In Context](https://mlsanity.com/paper/2506.00871/) (2025)
- [Enhancing Trajectory Prediction through Self-Supervised Waypoint Noise Prediction](https://mlsanity.com/paper/2312.09466/) (2023)
- [TrajAware: Graph Cross-Attention and Trajectory-Aware for Generalisable VANETs under Partial Observations](https://mlsanity.com/paper/2509.06665/) (2025)

## Abstract (arXiv)

Trajectory prediction is a crucial undertaking in understanding entity movement or human behavior from observed sequences. However, current methods often assume that the observed sequences are complete while ignoring the potential for missing values caused by object occlusion, scope limitation, sensor failure, etc. This limitation inevitably hinders the accuracy of trajectory prediction. To address this issue, our paper presents a unified framework, the Graph-based Conditional Variational Recurrent Neural Network (GC-VRNN), which can perform trajectory imputation and prediction simultaneously. Specifically, we introduce a novel Multi-Space Graph Neural Network (MS-GNN) that can extract spatial features from incomplete observations and leverage missing patterns. Additionally, we employ a Conditional VRNN with a specifically designed Temporal Decay (TD) module to capture temporal dependencies and temporal missing patterns in incomplete trajectories. The inclusion of the TD module allows for valuable information to be conveyed through the temporal flow. We also curate and benchmark three practical datasets for the joint problem of trajectory imputation and prediction. Extensive experiments verify the exceptional performance of our proposed method. As far as we know, this is the first work to address the lack of benchmarks and techniques for trajectory imputation and prediction in a unified manner.