# Learning to Extract Motion from Videos in Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1601.07532 · HTML: https://mlsanity.com/paper/1601.07532/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2016.

## Built on by (most cited first, 5 of 5)

- [ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets](https://mlsanity.com/paper/2209.00613/) (2022)
- [DDCNet: Deep Dilated Convolutional Neural Network for Dense Prediction](https://mlsanity.com/paper/2107.04715/) (2021)
- [SSTM: Spatiotemporal Recurrent Transformers for Multi-frame Optical Flow Estimation](https://mlsanity.com/paper/2304.14418/) (2023)
- [Learning Omnidirectional Flow in 360-degree Video via Siamese Representation](https://mlsanity.com/paper/2208.03620/) (2022)
- [Optical Flow Estimation in 360^circ Videos: Dataset, Model and Application](https://mlsanity.com/paper/2301.11880/) (2023)

## Abstract (arXiv)

This paper shows how to extract dense optical flow from videos with a convolutional neural network (CNN). The proposed model constitutes a potential building block for deeper architectures to allow using motion without resorting to an external algorithm, eg for recognition in videos. We derive our network architecture from signal processing principles to provide desired invariances to image contrast, phase and texture. We constrain weights within the network to enforce strict rotation invariance and substantially reduce the number of parameters to learn. We demonstrate end-to-end training on only 8 sequences of the Middlebury dataset, orders of magnitude less than competing CNN-based motion estimation methods, and obtain comparable performance to classical methods on the Middlebury benchmark. Importantly, our method outputs a distributed representation of motion that allows representing multiple, transparent motions, and dynamic textures. Our contributions on network design and rotation invariance offer insights nonspecific to motion estimation.