# AutoFlow: Learning a Better Training Set for Optical Flow

arXiv: https://arxiv.org/abs/2104.14544 · HTML: https://mlsanity.com/paper/2104.14544/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 54 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Domain-invariant Stereo Matching Networks](https://mlsanity.com/paper/1911.13287/) (2019)
- [Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation](https://mlsanity.com/paper/1809.05571/) (2018)
- [Hierarchical Discrete Distribution Decomposition for Match Density Estimation](https://mlsanity.com/paper/1812.06264/) (2018)
- [What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?](https://mlsanity.com/paper/1801.06397/) (2018)
- [Improving 3D Object Detection through Progressive Population Based Augmentation](https://mlsanity.com/paper/2004.00831/) (2020)
- [Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation](https://mlsanity.com/paper/1804.04259/) (2018)

## Built on by (most cited first, 30 of 102)

- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement](https://mlsanity.com/paper/2306.08637/) (2023)
- [PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking](https://mlsanity.com/paper/2307.15055/) (2023)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [Unifying Flow, Stereo and Depth Estimation](https://mlsanity.com/paper/2211.05783/) (2022)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation](https://mlsanity.com/paper/2203.11483/) (2022)
- [Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo](https://mlsanity.com/paper/2303.01943/) (2023)
- [CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion](https://mlsanity.com/paper/2210.10716/) (2022)
- [DynamicStereo: Consistent Dynamic Depth from Stereo Videos](https://mlsanity.com/paper/2305.02296/) (2023)
- [The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation](https://mlsanity.com/paper/2306.01923/) (2023)
- [FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation](https://mlsanity.com/paper/2303.01237/) (2023)
- [GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation](https://mlsanity.com/paper/2403.12365/) (2024)
- [Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos](https://mlsanity.com/paper/2412.09621/) (2024)
- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation](https://mlsanity.com/paper/2303.08340/) (2023)
- [PDC-Net+: Enhanced Probabilistic Dense Correspondence Network](https://mlsanity.com/paper/2109.13912/) (2021)
- [Deep Equilibrium Optical Flow Estimation](https://mlsanity.com/paper/2204.08442/) (2022)
- [AllTracker: Efficient Dense Point Tracking at High Resolution](https://mlsanity.com/paper/2506.07310/) (2025)
- [Residual Local Feature Network for Efficient Super-Resolution](https://mlsanity.com/paper/2205.07514/) (2022)
- [Dense Continuous-Time Optical Flow from Events and Frames](https://mlsanity.com/paper/2203.13674/) (2022)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [Learning Pixel Trajectories with Multiscale Contrastive Random Walks](https://mlsanity.com/paper/2201.08379/) (2022)
- [Rethinking Optical Flow from Geometric Matching Consistent Perspective](https://mlsanity.com/paper/2303.08384/) (2023)
- [Revisiting RCAN: Improved Training for Image Super-Resolution](https://mlsanity.com/paper/2201.11279/) (2022)
- [Disentangling Architecture and Training for Optical Flow](https://mlsanity.com/paper/2203.10712/) (2022)

## Abstract (arXiv)

Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications. To automate the process, we present AutoFlow, a simple and effective method to render training data for optical flow that optimizes the performance of a model on a target dataset. AutoFlow takes a layered approach to render synthetic data, where the motion, shape, and appearance of each layer are controlled by learnable hyperparameters. Experimental results show that AutoFlow achieves state-of-the-art accuracy in pre-training both PWC-Net and RAFT. Our code and data are available at https://autoflow-google.github.io .