# Semantic Prediction: Which One Should Come First, Recognition or Prediction?

arXiv: https://arxiv.org/abs/2110.02829 · HTML: https://mlsanity.com/paper/2110.02829/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 10 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 8 of 8)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Predicting Deeper into the Future of Semantic Segmentation](https://mlsanity.com/paper/1703.07684/) (2017)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [Real-Time Multi-View 3D Human Pose Estimation using Semantic Feedback to Smart Edge Sensors](https://mlsanity.com/paper/2106.14729/) (2021)
- [Local Frequency Domain Transformer Networks for Video Prediction](https://mlsanity.com/paper/2105.04637/) (2021)
- [Frequency Domain Transformer Networks for Video Prediction](https://mlsanity.com/paper/1903.00271/) (2019)
- [Motion Segmentation using Frequency Domain Transformer Networks](https://mlsanity.com/paper/2004.08638/) (2020)
- [Location Dependency in Video Prediction](https://mlsanity.com/paper/1810.04937/) (2018)

## Built on by (most cited first, 1 of 1)

- [Computational limits to the legibility of the imaged human brain](https://mlsanity.com/paper/2309.07096/) (2023)

## Abstract (arXiv)

The ultimate goal of video prediction is not forecasting future pixel-values given some previous frames. Rather, the end goal of video prediction is to discover valuable internal representations from the vast amount of available unlabeled video data in a self-supervised fashion for downstream tasks. One of the primary downstream tasks is interpreting the scene's semantic composition and using it for decision-making. For example, by predicting human movements, an observer can anticipate human activities and collaborate in a shared workspace. There are two main ways to achieve the same outcome, given a pre-trained video prediction and pre-trained semantic extraction model; one can first apply predictions and then extract semantics or first extract semantics and then predict. We investigate these configurations using the Local Frequency Domain Transformer Network (LFDTN) as the video prediction model and U-Net as the semantic extraction model on synthetic and real datasets.