# Learning Motion Patterns in Videos

arXiv: https://arxiv.org/abs/1612.07217 · HTML: https://mlsanity.com/paper/1612.07217/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2016.

## Built on by (most cited first, 30 of 53)

- [Occluded Video Instance Segmentation: A Benchmark](https://mlsanity.com/paper/2102.01558/) (2021)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Full-Duplex Strategy for Video Object Segmentation](https://mlsanity.com/paper/2108.03151/) (2021)
- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Implicit Motion Handling for Video Camouflaged Object Detection](https://mlsanity.com/paper/2203.07363/) (2022)
- [NeuralDiff: Segmenting 3D objects that move in egocentric videos](https://mlsanity.com/paper/2110.09936/) (2021)
- [Learning to Segment Rigid Motions from Two Frames](https://mlsanity.com/paper/2101.03694/) (2021)
- [The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos](https://mlsanity.com/paper/2111.06394/) (2021)
- [Improving Audio-Visual Segmentation with Bidirectional Generation](https://mlsanity.com/paper/2308.08288/) (2023)
- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [EM-driven unsupervised learning for efficient motion segmentation](https://mlsanity.com/paper/2201.02074/) (2022)
- [Unsupervised Video Object Segmentation via Prototype Memory Network](https://mlsanity.com/paper/2209.03712/) (2022)
- [Unsupervised Object Learning via Common Fate](https://mlsanity.com/paper/2110.06562/) (2021)
- [Iteratively Selecting an Easy Reference Frame Makes Unsupervised Video Object Segmentation Easier](https://mlsanity.com/paper/2112.12402/) (2021)
- [Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentation](https://mlsanity.com/paper/2104.04782/) (2021)
- [Guided Interactive Video Object Segmentation Using Reliability-Based Attention Maps](https://mlsanity.com/paper/2104.10386/) (2021)
- [Multi-Source Fusion and Automatic Predictor Selection for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2108.05076/) (2021)
- [Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning](https://mlsanity.com/paper/2109.13499/) (2021)
- [Multi-Object Discovery by Low-Dimensional Object Motion](https://mlsanity.com/paper/2307.08027/) (2023)
- [A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2209.09341/) (2022)
- [Motion-inductive Self-supervised Object Discovery in Videos](https://mlsanity.com/paper/2210.00221/) (2022)
- [UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model](https://mlsanity.com/paper/2305.12659/) (2023)
- [Spatiotemporal Data Mining: A Survey on Challenges and Open Problems](https://mlsanity.com/paper/2103.17128/) (2021)
- [Improving Video Instance Segmentation via Temporal Pyramid Routing](https://mlsanity.com/paper/2107.13155/) (2021)
- [Video Salient Object Detection via Contrastive Features and Attention Modules](https://mlsanity.com/paper/2111.02368/) (2021)
- [L4P: Towards Unified Low-Level 4D Vision Perception](https://mlsanity.com/paper/2502.13078/) (2025)
- [Appearance-Based Refinement for Object-Centric Motion Segmentation](https://mlsanity.com/paper/2312.11463/) (2023)

## Abstract (arXiv)

The problem of determining whether an object is in motion, irrespective of camera motion, is far from being solved. We address this challenging task by learning motion patterns in videos. The core of our approach is a fully convolutional network, which is learned entirely from synthetic video sequences, and their ground-truth optical flow and motion segmentation. This encoder-decoder style architecture first learns a coarse representation of the optical flow field features, and then refines it iteratively to produce motion labels at the original high-resolution. We further improve this labeling with an objectness map and a conditional random field, to account for errors in optical flow, and also to focus on moving "things" rather than "stuff". The output label of each pixel denotes whether it has undergone independent motion, i.e., irrespective of camera motion. We demonstrate the benefits of this learning framework on the moving object segmentation task, where the goal is to segment all objects in motion. Our approach outperforms the top method on the recently released DAVIS benchmark dataset, comprising real-world sequences, by 5.6%. We also evaluate on the Berkeley motion segmentation database, achieving state-of-the-art results.