# Learning To Segment Dominant Object Motion From Watching Videos

arXiv: https://arxiv.org/abs/2111.14160 · HTML: https://mlsanity.com/paper/2111.14160/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 28 Nov 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [Learning Video Object Segmentation from Static Images](https://mlsanity.com/paper/1612.02646/) (2016)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks](https://mlsanity.com/paper/2001.06810/) (2020)
- [MAST: A Memory-Augmented Self-supervised Tracker](https://mlsanity.com/paper/2002.07793/) (2020)
- [Video Object Segmentation Without Temporal Information](https://mlsanity.com/paper/1709.06031/) (2017)
- [FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos](https://mlsanity.com/paper/1701.05384/) (2017)
- [Deep Neural Decision Trees](https://mlsanity.com/paper/1806.06988/) (2018)
- [Video Object Segmentation with Joint Re-identification and Attention-Aware Mask Propagation](https://mlsanity.com/paper/1803.04242/) (2018)
- [Making a Case for 3D Convolutions for Object Segmentation in Videos](https://mlsanity.com/paper/2008.11516/) (2020)
- [Anchor Diffusion for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/1910.10895/) (2019)
- [Learning Video Object Segmentation from Unlabeled Videos](https://mlsanity.com/paper/2003.05020/) (2020)
- [Learning to Segment Moving Objects](https://mlsanity.com/paper/1712.01127/) (2017)
- [Make One-Shot Video Object Segmentation Efficient Again](https://mlsanity.com/paper/2012.01866/) (2020)
- [Optical Flow with Semantic Segmentation and Localized Layers](https://mlsanity.com/paper/1603.03911/) (2016)
- [Unsupervised Online Video Object Segmentation with Motion Property Understanding](https://mlsanity.com/paper/1810.03783/) (2018)
- [Shape Interaction Matrix Revisited and Robustified: Efficient Subspace Clustering with Corrupted and Incomplete Data](https://mlsanity.com/paper/1509.02649/) (2015)
- [Dual Rectified Linear Units (DReLUs): A Replacement for Tanh Activation Functions in Quasi-Recurrent Neural Networks](https://mlsanity.com/paper/1707.08214/) (2017)
- [Robust Motion Segmentation from Pairwise Matches](https://mlsanity.com/paper/1905.09043/) (2019)
- [Layered Optical Flow Estimation Using a Deep Neural Network with a Soft Mask](https://mlsanity.com/paper/1805.03596/) (2018)

## Built on by (most cited first, 1 of 1)

- [EV-LayerSegNet: Self-supervised Motion Segmentation using Event Cameras](https://mlsanity.com/paper/2506.06596/) (2025)

## Abstract (arXiv)

Existing deep learning based unsupervised video object segmentation methods still rely on ground-truth segmentation masks to train. Unsupervised in this context only means that no annotated frames are used during inference. As obtaining ground-truth segmentation masks for real image scenes is a laborious task, we envision a simple framework for dominant moving object segmentation that neither requires annotated data to train nor relies on saliency priors or pre-trained optical flow maps. Inspired by a layered image representation, we introduce a technique to group pixel regions according to their affine parametric motion. This enables our network to learn segmentation of the dominant foreground object using only RGB image pairs as input for both training and inference. We establish a baseline for this novel task using a new MovingCars dataset and show competitive performance against recent methods that require annotated masks to train.