# Deep Learning for Vision-based Prediction: A Survey

arXiv: https://arxiv.org/abs/2007.00095 · HTML: https://mlsanity.com/paper/2007.00095/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2020.

## Built on by (most cited first, 19 of 19)

- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Predicting the Future from First Person (Egocentric) Vision: A Survey](https://mlsanity.com/paper/2107.13411/) (2021)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)
- [Pedestrian Simulation: A Review](https://mlsanity.com/paper/2102.03289/) (2021)
- [Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery](https://mlsanity.com/paper/2402.01974/) (2024)
- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [Pedestrian Trajectory Prediction via Spatial Interaction Transformer Network](https://mlsanity.com/paper/2112.06624/) (2021)
- [Scenario-based Evaluation of Prediction Models for Automated Vehicles](https://mlsanity.com/paper/2210.06553/) (2022)
- [Generating Synthetic Training Data for Deep Learning-Based UAV Trajectory Prediction](https://mlsanity.com/paper/2107.00422/) (2021)
- [Handling Missing Observations with an RNN-based Prediction-Update Cycle](https://mlsanity.com/paper/2103.11747/) (2021)
- [MissFormer: (In-)attention-based handling of missing observations for trajectory filtering and prediction](https://mlsanity.com/paper/2106.16009/) (2021)
- [ROME: Robust Multi-Modal Density Estimator](https://mlsanity.com/paper/2401.10566/) (2024)
- [Lifelong Learning of Video Diffusion Models From a Single Video Stream](https://mlsanity.com/paper/2406.04814/) (2024)
- [Robustness Benchmark of Road User Trajectory Prediction Models for Automated Driving](https://mlsanity.com/paper/2304.01895/) (2023)
- [Dynamic Risk Assessment for Autonomous Vehicles from Spatio-Temporal Probabilistic Occupancy Heatmaps](https://mlsanity.com/paper/2501.16480/) (2025)
- [Motion Prediction Performance Analysis for Autonomous Driving Systems and the Effects of Tracking Noise](https://mlsanity.com/paper/2104.08368/) (2021)
- [Transfer Learning for Estimation of Pendubot Angular Position Using Deep Neural Networks](https://mlsanity.com/paper/2201.12649/) (2022)

## Abstract (arXiv)

Vision-based prediction algorithms have a wide range of applications including autonomous driving, surveillance, human-robot interaction, weather prediction. The objective of this paper is to provide an overview of the field in the past five years with a particular focus on deep learning approaches. For this purpose, we categorize these algorithms into video prediction, action prediction, trajectory prediction, body motion prediction, and other prediction applications. For each category, we highlight the common architectures, training methods and types of data used. In addition, we discuss the common evaluation metrics and datasets used for vision-based prediction tasks. A database of all the information presented in this survey including, cross-referenced according to papers, datasets and metrics, can be found online at https://github.com/aras62/vision-based-prediction.