# Local Frequency Domain Transformer Networks for Video Prediction

arXiv: https://arxiv.org/abs/2105.04637 · HTML: https://mlsanity.com/paper/2105.04637/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 10 of 10)

- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Semi-Supervised Learning with Ladder Networks](https://mlsanity.com/paper/1507.02672/) (2015)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning](https://mlsanity.com/paper/1804.06300/) (2018)
- [Transformation-Based Models of Video Sequences](https://mlsanity.com/paper/1701.08435/) (2017)
- [Frequency Domain Transformer Networks for Video Prediction](https://mlsanity.com/paper/1903.00271/) (2019)
- [A Neurally-Inspired Hierarchical Prediction Network for Spatiotemporal Sequence Learning and Prediction](https://mlsanity.com/paper/1901.09002/) (2019)
- [Motion Segmentation using Frequency Domain Transformer Networks](https://mlsanity.com/paper/2004.08638/) (2020)
- [Video Ladder Networks](https://mlsanity.com/paper/1612.01756/) (2016)
- [Location Dependency in Video Prediction](https://mlsanity.com/paper/1810.04937/) (2018)

## Built on by (most cited first, 9 of 9)

- [Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactions](https://mlsanity.com/paper/2302.11850/) (2023)
- [DiJiang: Efficient Large Language Models through Compact Kernelization](https://mlsanity.com/paper/2403.19928/) (2024)
- [MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networks](https://mlsanity.com/paper/2203.09303/) (2022)
- [Unsupervised Image Decomposition with Phase-Correlation Networks](https://mlsanity.com/paper/2110.03473/) (2021)
- [The Power of Next-Frame Prediction for Learning Physical Laws](https://mlsanity.com/paper/2405.17450/) (2024)
- [Semantic Prediction: Which One Should Come First, Recognition or Prediction?](https://mlsanity.com/paper/2110.02829/) (2021)
- [Fourier-based Video Prediction through Relational Object Motion](https://mlsanity.com/paper/2110.05881/) (2021)
- [State-space Decomposition Model for Video Prediction Considering Long-term Motion Trend](https://mlsanity.com/paper/2404.11576/) (2024)
- [VideoPCDNet: Video Parsing and Prediction with Phase Correlation Networks](https://mlsanity.com/paper/2506.19621/) (2025)

## Abstract (arXiv)

Video prediction is commonly referred to as forecasting future frames of a video sequence provided several past frames thereof. It remains a challenging domain as visual scenes evolve according to complex underlying dynamics, such as the camera's egocentric motion or the distinct motility per individual object viewed. These are mostly hidden from the observer and manifest as often highly non-linear transformations between consecutive video frames. Therefore, video prediction is of interest not only in anticipating visual changes in the real world but has, above all, emerged as an unsupervised learning rule targeting the formation and dynamics of the observed environment. Many of the deep learning-based state-of-the-art models for video prediction utilize some form of recurrent layers like Long Short-Term Memory (LSTMs) or Gated Recurrent Units (GRUs) at the core of their models. Although these models can predict the future frames, they rely entirely on these recurrent structures to simultaneously perform three distinct tasks: extracting transformations, projecting them into the future, and transforming the current frame. In order to completely interpret the formed internal representations, it is crucial to disentangle these tasks. This paper proposes a fully differentiable building block that can perform all of those tasks separately while maintaining interpretability. We derive the relevant theoretical foundations and showcase results on synthetic as well as real data. We demonstrate that our method is readily extended to perform motion segmentation and account for the scene's composition, and learns to produce reliable predictions in an entirely interpretable manner by only observing unlabeled video data.