# Deep Learning for Robust Motion Segmentation with Non-Static Cameras

arXiv: https://arxiv.org/abs/2102.10929 · HTML: https://mlsanity.com/paper/2102.10929/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 63 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Object Detectors Emerge in Deep Scene CNNs](https://mlsanity.com/paper/1412.6856/) (2014)
- [Salient Object Detection: A Survey](https://mlsanity.com/paper/1411.5878/) (2014)
- [Fast Optical Flow using Dense Inverse Search](https://mlsanity.com/paper/1603.03590/) (2016)
- [Background Subtraction in Real Applications: Challenges, Current Models and Future Directions](https://mlsanity.com/paper/1901.03577/) (2019)
- [A Survey on Deep Learning-based Architectures for Semantic Segmentation on 2D images](https://mlsanity.com/paper/1912.10230/) (2019)
- [Learning Multi-scale Features for Foreground Segmentation](https://mlsanity.com/paper/1808.01477/) (2018)
- [Spatiotemporal CNN for Video Object Segmentation](https://mlsanity.com/paper/1904.02363/) (2019)
- [BSUV-Net: A Fully-Convolutional Neural Network for Background Subtraction of Unseen Videos](https://mlsanity.com/paper/1907.11371/) (2019)
- [What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?](https://mlsanity.com/paper/1705.07049/) (2017)
- [Foreground Segmentation Using a Triplet Convolutional Neural Network for Multiscale Feature Encoding](https://mlsanity.com/paper/1801.02225/) (2018)
- [Flying Objects Detection from a Single Moving Camera](https://mlsanity.com/paper/1411.7715/) (2014)
- [3DFR: A Swift 3D Feature Reductionist Framework for Scene Independent Change Detection](https://mlsanity.com/paper/1912.11891/) (2019)
- [ClusterNet: Detecting Small Objects in Large Scenes by Exploiting Spatio-Temporal Information](https://mlsanity.com/paper/1704.02694/) (2017)
- [A Foreground Inference Network for Video Surveillance Using Multi-View Receptive Field](https://mlsanity.com/paper/1801.06593/) (2018)

## Built on by (most cited first, 5 of 5)

- [Segment Any Motion in Videos](https://mlsanity.com/paper/2503.22268/) (2025)
- [Motion Segmentation from a Moving Monocular Camera](https://mlsanity.com/paper/2309.13772/) (2023)
- [Zero-Shot Monocular Motion Segmentation in the Wild by Combining Deep Learning with Geometric Motion Model Fusion](https://mlsanity.com/paper/2405.01723/) (2024)
- [Dense Monocular Motion Segmentation Using Optical Flow and Pseudo Depth Map: A Zero-Shot Approach](https://mlsanity.com/paper/2406.18837/) (2024)
- [SUMO: Segment and Track Any Motion with Nonlinear State Space Models](https://mlsanity.com/paper/2606.29861/) (2026)

## Abstract (arXiv)

This work proposes a new end-to-end DCNN based approach for motion segmentation, especially for video sequences captured with such non-static cameras, called MOSNET. While other approaches focus on spatial or temporal context only, the proposed approach uses 3D convolutions as a key technology to factor in, spatio-temporal features in cohesive video frames. This is done by capturing temporal information in features with a low and also with a high level of abstraction. The lean network architecture with about 21k trainable parameters is mainly based on a pre-trained VGG-16 network. The MOSNET uses a new feature map fusion technique, which enables the network to focus on the appropriate level of abstraction, resolution, and the appropriate size of the receptive field regarding the input. Furthermore, the end-to-end deep learning based approach can be extended by feature based image alignment as a pre-processing step, which brings a gain in performance for some scenes. Evaluating the end-to-end deep learning based MOSNET network in a scene independent manner leads to an overall F-measure of 0.803 on the CDNet2014 dataset. A small temporal window of five input frames, without the need of any initialization is used to obtain this result. Therefore the network is able to perform well on scenes captured with non-static cameras where the image content changes significantly during the scene. In order to get robust results in scenes captured with a moving camera, feature based image alignment can implemented as pre-processing step. The MOSNET combined with pre-processing leads to an F-measure of 0.685 when cross-evaluating with a relabeled LASIESTA dataset, which underpins the capability generalise of the MOSNET.