# Towards Robust Human Trajectory Prediction in Raw Videos

arXiv: https://arxiv.org/abs/2108.08259 · HTML: https://mlsanity.com/paper/2108.08259/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 18 Aug 2021.

## Builds on (most cited first, 28 of 28)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [PnPNet: End-to-End Perception and Prediction with Tracking in the Loop](https://mlsanity.com/paper/2005.14711/) (2020)
- [MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking](https://mlsanity.com/paper/1504.01942/) (2015)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data](https://mlsanity.com/paper/1910.08233/) (2019)
- [What the Constant Velocity Model Can Teach Us About Pedestrian Motion Prediction](https://mlsanity.com/paper/1903.07933/) (2019)
- [Conditional Generative Neural System for Probabilistic Trajectory Prediction](https://mlsanity.com/paper/1905.01631/) (2019)
- [Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty](https://mlsanity.com/paper/1711.09026/) (2017)
- [Future Person Localization in First-Person Videos](https://mlsanity.com/paper/1711.11217/) (2017)
- [End-to-end Contextual Perception and Prediction with Interaction Transformer](https://mlsanity.com/paper/2008.05927/) (2020)
- [LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusion](https://mlsanity.com/paper/2010.00731/) (2020)
- [MX-LSTM: mixing tracklets and vislets to jointly forecast trajectories and head poses](https://mlsanity.com/paper/1805.00652/) (2018)
- [Tracking by Prediction: A Deep Generative Model for Mutli-Person localisation and Tracking](https://mlsanity.com/paper/1803.03347/) (2018)
- [AutoTrajectory: Label-free Trajectory Extraction and Prediction from Videos using Dynamic Points](https://mlsanity.com/paper/2007.05719/) (2020)

## Built on by (most cited first, 7 of 7)

- [MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation](https://mlsanity.com/paper/2110.09481/) (2021)
- [T2FPV: Dataset and Method for Correcting First-Person View Errors in Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2209.11294/) (2022)
- [DESTINE: Dynamic Goal Queries with Temporal Transductive Alignment for Trajectory Prediction](https://mlsanity.com/paper/2310.07438/) (2023)
- [StreamMOTP: Streaming and Unified Framework for Joint 3D Multi-Object Tracking and Trajectory Prediction](https://mlsanity.com/paper/2406.19844/) (2024)
- [RealTraj: Towards Real-World Pedestrian Trajectory Forecasting](https://mlsanity.com/paper/2411.17376/) (2024)
- [DuoSpaceNet: Leveraging Both Bird's-Eye-View and Perspective View Representations for 3D Object Detection](https://mlsanity.com/paper/2405.10577/) (2024)
- [Uncertainty-Aware Trajectory Forecasting from Imperfect Tracking](https://mlsanity.com/paper/2608.30899/) (2026)

## Abstract (arXiv)

Human trajectory prediction has received increased attention lately due to its importance in applications such as autonomous vehicles and indoor robots. However, most existing methods make predictions based on human-labeled trajectories and ignore the errors and noises in detection and tracking. In this paper, we study the problem of human trajectory forecasting in raw videos, and show that the prediction accuracy can be severely affected by various types of tracking errors. Accordingly, we propose a simple yet effective strategy to correct the tracking failures by enforcing prediction consistency over time. The proposed "re-tracking" algorithm can be applied to any existing tracking and prediction pipelines. Experiments on public benchmark datasets demonstrate that the proposed method can improve both tracking and prediction performance in challenging real-world scenarios. The code and data are available at https://git.io/retracking-prediction.