# Self-Supervised Monocular Scene Decomposition and Depth Estimation

arXiv: https://arxiv.org/abs/2110.11275 · HTML: https://mlsanity.com/paper/2110.11275/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 21 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos](https://mlsanity.com/paper/1811.06152/) (2018)
- [Learning Depth from Monocular Videos using Direct Methods](https://mlsanity.com/paper/1712.00175/) (2017)
- [DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency](https://mlsanity.com/paper/1809.01649/) (2018)
- [Semantically-Guided Representation Learning for Self-Supervised Monocular Depth](https://mlsanity.com/paper/2002.12319/) (2020)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding](https://mlsanity.com/paper/1810.06125/) (2018)
- [Unsupervised Monocular Depth Learning in Dynamic Scenes](https://mlsanity.com/paper/2010.16404/) (2020)
- [SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks](https://mlsanity.com/paper/1606.02378/) (2016)
- [Unsupervised part representation by Flow Capsules](https://mlsanity.com/paper/2011.13920/) (2020)
- [Learning Independent Object Motion from Unlabelled Stereoscopic Videos](https://mlsanity.com/paper/1901.01971/) (2019)
- [Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint](https://mlsanity.com/paper/1604.02885/) (2016)

## Built on by (most cited first, 8 of 8)

- [MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer](https://mlsanity.com/paper/2208.03543/) (2022)
- [Multi-Object Discovery by Low-Dimensional Object Motion](https://mlsanity.com/paper/2307.08027/) (2023)
- [Stochastic Video Prediction with Structure and Motion](https://mlsanity.com/paper/2203.10528/) (2022)
- [Instance-aware multi-object self-supervision for monocular depth prediction](https://mlsanity.com/paper/2203.00809/) (2022)
- [Dyna-DM: Dynamic Object-aware Self-supervised Monocular Depth Maps](https://mlsanity.com/paper/2206.03799/) (2022)
- [STDepthFormer: Predicting Spatio-temporal Depth from Video with a Self-supervised Transformer Model](https://mlsanity.com/paper/2303.01196/) (2023)
- [Stochastic Future Prediction in Real World Driving Scenarios](https://mlsanity.com/paper/2209.10693/) (2022)
- [DepthP+P: Metric Accurate Monocular Depth Estimation using Planar and Parallax](https://mlsanity.com/paper/2301.02092/) (2023)

## Abstract (arXiv)

Self-supervised monocular depth estimation approaches either ignore independently moving objects in the scene or need a separate segmentation step to identify them. We propose MonoDepthSeg to jointly estimate depth and segment moving objects from monocular video without using any ground-truth labels. We decompose the scene into a fixed number of components where each component corresponds to a region on the image with its own transformation matrix representing its motion. We estimate both the mask and the motion of each component efficiently with a shared encoder. We evaluate our method on three driving datasets and show that our model clearly improves depth estimation while decomposing the scene into separately moving components.