# Autoencoder-based background reconstruction and foreground segmentation with background noise estimation

arXiv: https://arxiv.org/abs/2112.08001 · HTML: https://mlsanity.com/paper/2112.08001/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 81 references that are in the map.
- First posted on arXiv 15 Dec 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation](https://mlsanity.com/paper/2109.09881/) (2021)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019)
- [ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/2111.10265/) (2021)
- [ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking](https://mlsanity.com/paper/1804.08018/) (2018)
- [CoordiNet: uncertainty-aware pose regressor for reliable vehicle localization](https://mlsanity.com/paper/2103.10796/) (2021)
- [Unsupervised object-centric video generation and decomposition in 3D](https://mlsanity.com/paper/2007.06705/) (2020)
- [Background Subtraction in Real Applications: Challenges, Current Models and Future Directions](https://mlsanity.com/paper/1901.03577/) (2019)
- [Learning Multi-scale Features for Foreground Segmentation](https://mlsanity.com/paper/1808.01477/) (2018)
- [Foreground Segmentation Using a Triplet Convolutional Neural Network for Multiscale Feature Encoding](https://mlsanity.com/paper/1801.02225/) (2018)
- [APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation](https://mlsanity.com/paper/2105.14895/) (2021)
- [BSUV-Net 2.0: Spatio-Temporal Data Augmentations for Video-Agnostic Supervised Background Subtraction](https://mlsanity.com/paper/2101.09585/) (2021)
- [GraphBGS: Background Subtraction via Recovery of Graph Signals](https://mlsanity.com/paper/2001.06404/) (2020)
- [An Empirical Review of Deep Learning Frameworks for Change Detection: Model Design, Experimental Frameworks, Challenges and Research Needs](https://mlsanity.com/paper/2105.01342/) (2021)
- [Unsupervised Deep Context Prediction for Background Foreground Separation](https://mlsanity.com/paper/1805.07903/) (2018)
- [WisenetMD: Motion Detection Using Dynamic Background Region Analysis](https://mlsanity.com/paper/1805.09277/) (2018)
- [How Far Can You Get By Combining Change Detection Algorithms?](https://mlsanity.com/paper/1505.02921/) (2015)
- [Background Subtraction with Real-time Semantic Segmentation](https://mlsanity.com/paper/1811.10020/) (2018)
- [Background Subtraction via Generalized Fused Lasso Foreground Modeling](https://mlsanity.com/paper/1504.03707/) (2015)
- [G-LBM:Generative Low-dimensional Background Model Estimation from Video Sequences](https://mlsanity.com/paper/2003.07335/) (2020)
- [DeepPBM: Deep Probabilistic Background Model Estimation from Video Sequences](https://mlsanity.com/paper/1902.00820/) (2019)

## Built on by (most cited first, 7 of 7)

- [Unsupervised Multi-object Segmentation Using Attention and Soft-argmax](https://mlsanity.com/paper/2205.13271/) (2022)
- [ZBS: Zero-shot Background Subtraction via Instance-level Background Modeling and Foreground Selection](https://mlsanity.com/paper/2303.14679/) (2023)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [Learning Temporal Distribution and Spatial Correlation Towards Universal Moving Object Segmentation](https://mlsanity.com/paper/2304.09949/) (2023)
- [Application Of ADNN For Background Subtraction In Smart Surveillance System](https://mlsanity.com/paper/2301.00264/) (2022)
- [Improving Prediction Accuracy of Semantic Segmentation Methods Using Convolutional Autoencoder Based Pre-processing Layers](https://mlsanity.com/paper/2404.12718/) (2024)
- [Quaternion-Based Robust PCA for Efficient Moving Target Detection and Background Recovery in Color Videos](https://mlsanity.com/paper/2507.19730/) (2025)

## Abstract (arXiv)

Even after decades of research, dynamic scene background reconstruction and foreground object segmentation are still considered as open problems due various challenges such as illumination changes, camera movements, or background noise caused by air turbulence or moving trees. We propose in this paper to model the background of a frame sequence as a low dimensional manifold using an autoencoder and compare the reconstructed background provided by this autoencoder with the original image to compute the foreground/background segmentation masks. The main novelty of the proposed model is that the autoencoder is also trained to predict the background noise, which allows to compute for each frame a pixel-dependent threshold to perform the foreground segmentation. Although the proposed model does not use any temporal or motion information, it exceeds the state of the art for unsupervised background subtraction on the CDnet 2014 and LASIESTA datasets, with a significant improvement on videos where the camera is moving. It is also able to perform background reconstruction on some non-video image datasets.