# Bootstrapped Self-Supervised Training with Monocular Video for Semantic Segmentation and Depth Estimation

arXiv: https://arxiv.org/abs/2103.11031 · HTML: https://mlsanity.com/paper/2103.11031/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 19 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)
- [Learning Depth from Monocular Videos using Direct Methods](https://mlsanity.com/paper/1712.00175/) (2017)
- [Learning Features by Watching Objects Move](https://mlsanity.com/paper/1612.06370/) (2016)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [Self-Supervised Model Adaptation for Multimodal Semantic Segmentation](https://mlsanity.com/paper/1808.03833/) (2018)
- [Semantic Video CNNs through Representation Warping](https://mlsanity.com/paper/1708.03088/) (2017)
- [SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded Scene Representations](https://mlsanity.com/paper/1903.06482/) (2019)
- [Efficient Video Semantic Segmentation with Labels Propagation and Refinement](https://mlsanity.com/paper/1912.11844/) (2019)
- [Mix-and-Match Tuning for Self-Supervised Semantic Segmentation](https://mlsanity.com/paper/1712.00661/) (2017)
- [Semantic Segmentation of Video Sequences with Convolutional LSTMs](https://mlsanity.com/paper/1905.01058/) (2019)
- [Weakly-Supervised Semantic Segmentation using Motion Cues](https://mlsanity.com/paper/1603.07188/) (2016)

## Built on by (most cited first, 2 of 2)

- [A Threefold Review on Deep Semantic Segmentation: Efficiency-oriented, Temporal and Depth-aware design](https://mlsanity.com/paper/2303.04315/) (2023)
- [A Front-End for Dense Monocular SLAM using a Learned Outlier Mask Prior](https://mlsanity.com/paper/2104.00562/) (2021)

## Abstract (arXiv)

For a robot deployed in the world, it is desirable to have the ability of autonomous learning to improve its initial pre-set knowledge. We formalize this as a bootstrapped self-supervised learning problem where a system is initially bootstrapped with supervised training on a labeled dataset and we look for a self-supervised training method that can subsequently improve the system over the supervised training baseline using only unlabeled data. In this work, we leverage temporal consistency between frames in monocular video to perform this bootstrapped self-supervised training. We show that a well-trained state-of-the-art semantic segmentation network can be further improved through our method. In addition, we show that the bootstrapped self-supervised training framework can help a network learn depth estimation better than pure supervised training or self-supervised training.