# Multigrid Predictive Filter Flow for Unsupervised Learning on Videos

arXiv: https://arxiv.org/abs/1904.01693 · HTML: https://mlsanity.com/paper/1904.01693/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 4 of 4)

- [CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.01040/) (2023)
- [Comparing Correspondences: Video Prediction with Correspondence-wise Losses](https://mlsanity.com/paper/2104.09498/) (2021)
- [Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning](https://mlsanity.com/paper/2105.05838/) (2021)
- [Self-supervised Semantic Segmentation Grounded in Visual Concepts](https://mlsanity.com/paper/2203.13868/) (2022)

## Abstract (arXiv)

We introduce multigrid Predictive Filter Flow (mgPFF), a framework for unsupervised learning on videos. The mgPFF takes as input a pair of frames and outputs per-pixel filters to warp one frame to the other. Compared to optical flow used for warping frames, mgPFF is more powerful in modeling sub-pixel movement and dealing with corruption (e.g., motion blur). We develop a multigrid coarse-to-fine modeling strategy that avoids the requirement of learning large filters to capture large displacement. This allows us to train an extremely compact model (4.6MB) which operates in a progressive way over multiple resolutions with shared weights. We train mgPFF on unsupervised, free-form videos and show that mgPFF is able to not only estimate long-range flow for frame reconstruction and detect video shot transitions, but also readily amendable for video object segmentation and pose tracking, where it substantially outperforms the published state-of-the-art without bells and whistles. Moreover, owing to mgPFF's nature of per-pixel filter prediction, we have the unique opportunity to visualize how each pixel is evolving during solving these tasks, thus gaining better interpretability.