# Motion Segmentation using Frequency Domain Transformer Networks

arXiv: https://arxiv.org/abs/2004.08638 · HTML: https://mlsanity.com/paper/2004.08638/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networks](https://mlsanity.com/paper/2203.09303/) (2022)
- [Local Frequency Domain Transformer Networks for Video Prediction](https://mlsanity.com/paper/2105.04637/) (2021)
- [Unsupervised Image Decomposition with Phase-Correlation Networks](https://mlsanity.com/paper/2110.03473/) (2021)
- [Semantic Prediction: Which One Should Come First, Recognition or Prediction?](https://mlsanity.com/paper/2110.02829/) (2021)
- [VideoPCDNet: Video Parsing and Prediction with Phase Correlation Networks](https://mlsanity.com/paper/2506.19621/) (2025)

## Abstract (arXiv)

Self-supervised prediction is a powerful mechanism to learn representations that capture the underlying structure of the data. Despite recent progress, the self-supervised video prediction task is still challenging. One of the critical factors that make the task hard is motion segmentation, which is segmenting individual objects and the background and estimating their motion separately. In video prediction, the shape, appearance, and transformation of each object should be understood only by predicting the next frame in pixel space. To address this task, we propose a novel end-to-end learnable architecture that predicts the next frame by modeling foreground and background separately while simultaneously estimating and predicting the foreground motion using Frequency Domain Transformer Networks. Experimental evaluations show that this yields interpretable representations and that our approach can outperform some widely used video prediction methods like Video Ladder Network and Predictive Gated Pyramids on synthetic data.