# SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping

arXiv: https://arxiv.org/abs/2105.07014 · HTML: https://mlsanity.com/paper/2105.07014/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 14 May 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency](https://mlsanity.com/paper/1809.01649/) (2018)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [Accurate Optical Flow via Direct Cost Volume Processing](https://mlsanity.com/paper/1704.07325/) (2017)
- [DDFlow: Learning Optical Flow with Unlabeled Data Distillation](https://mlsanity.com/paper/1902.09145/) (2019)
- [Flow2Stereo: Effective Self-Supervised Learning of Optical Flow and Stereo Matching](https://mlsanity.com/paper/2004.02138/) (2020)
- [Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids](https://mlsanity.com/paper/1604.03513/) (2016)
- [Reversing the cycle: self-supervised deep stereo through enhanced monocular distillation](https://mlsanity.com/paper/2008.07130/) (2020)
- [Self-Supervised Learning for Stereo Matching with Self-Improving Ability](https://mlsanity.com/paper/1709.00930/) (2017)
- [Unsupervised Deep Epipolar Flow for Stationary or Dynamic Scenes](https://mlsanity.com/paper/1904.03848/) (2019)
- [ProFlow: Learning to Predict Optical Flow](https://mlsanity.com/paper/1806.00800/) (2018)

## Built on by (most cited first, 30 of 74)

- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [BootsTAP: Bootstrapped Training for Tracking-Any-Point](https://mlsanity.com/paper/2402.00847/) (2024)
- [FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation](https://mlsanity.com/paper/2303.01237/) (2023)
- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [Learning Dense and Continuous Optical Flow from an Event Camera](https://mlsanity.com/paper/2211.09078/) (2022)
- [MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features](https://mlsanity.com/paper/2307.12698/) (2023)
- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [Discovering Objects that Can Move](https://mlsanity.com/paper/2203.10159/) (2022)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [Learning Pixel Trajectories with Multiscale Contrastive Random Walks](https://mlsanity.com/paper/2201.08379/) (2022)
- [Disentangling Architecture and Training for Optical Flow](https://mlsanity.com/paper/2203.10712/) (2022)
- [Object Discovery from Motion-Guided Tokens](https://mlsanity.com/paper/2303.15555/) (2023)
- [Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation](https://mlsanity.com/paper/2407.10802/) (2024)
- [RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos](https://mlsanity.com/paper/2207.11075/) (2022)
- [RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes](https://mlsanity.com/paper/2303.04456/) (2023)
- [Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns](https://mlsanity.com/paper/2210.12148/) (2022)
- [Unifying (Machine) Vision via Counterfactual World Modeling](https://mlsanity.com/paper/2306.01828/) (2023)
- [Boosting Object Representation Learning via Motion and Object Continuity](https://mlsanity.com/paper/2211.09771/) (2022)
- [Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow](https://mlsanity.com/paper/2403.07432/) (2024)
- [Optical Flow Training under Limited Label Budget via Active Learning](https://mlsanity.com/paper/2203.05053/) (2022)
- [M-FUSE: Multi-frame Fusion for Scene Flow Estimation](https://mlsanity.com/paper/2207.05704/) (2022)
- [DistractFlow: Improving Optical Flow Estimation via Realistic Distractions and Pseudo-Labeling](https://mlsanity.com/paper/2303.14078/) (2023)
- [Refining Pre-Trained Motion Models](https://mlsanity.com/paper/2401.00850/) (2024)
- [OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation](https://mlsanity.com/paper/2403.18092/) (2024)
- [MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation](https://mlsanity.com/paper/2211.06018/) (2022)
- [Self-supervised AutoFlow](https://mlsanity.com/paper/2212.01762/) (2022)
- [GyroFlow+: Gyroscope-Guided Unsupervised Deep Homography and Optical Flow Learning](https://mlsanity.com/paper/2301.10018/) (2023)
- [Self-Supervised Any-Point Tracking by Contrastive Random Walks](https://mlsanity.com/paper/2409.16288/) (2024)
- [Scene-Centric Unsupervised Panoptic Segmentation](https://mlsanity.com/paper/2504.01955/) (2025)

## Abstract (arXiv)

We present SMURF, a method for unsupervised learning of optical flow that improves state of the art on all benchmarks by 36% to 40% (over the prior best method UFlow) and even outperforms several supervised approaches such as PWC-Net and FlowNet2. Our method integrates architecture improvements from supervised optical flow, i.e. the RAFT model, with new ideas for unsupervised learning that include a sequence-aware self-supervision loss, a technique for handling out-of-frame motion, and an approach for learning effectively from multi-frame video data while still only requiring two frames for inference.