# S3Net: Semantic-Aware Self-supervised Depth Estimation with Monocular Videos and Synthetic Data

arXiv: https://arxiv.org/abs/2007.14511 · HTML: https://mlsanity.com/paper/2007.14511/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2020.

## Built on by (most cited first, 12 of 12)

- [Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction](https://mlsanity.com/paper/2103.12091/) (2021)
- [Monocular Depth Estimation through Virtual-world Supervision and Real-world SfM Self-Supervision](https://mlsanity.com/paper/2103.12209/) (2021)
- [Self-distilled Feature Aggregation for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2209.07088/) (2022)
- [Two-in-One Depth: Bridging the Gap Between Monocular and Binocular Self-supervised Depth Estimation](https://mlsanity.com/paper/2309.00933/) (2023)
- [Learn to Adapt for Monocular Depth Estimation](https://mlsanity.com/paper/2203.14005/) (2022)
- [How Far Can I Go ? : A Self-Supervised Approach for Deterministic Video Depth Forecasting](https://mlsanity.com/paper/2207.00506/) (2022)
- [ADAADepth: Adapting Data Augmentation and Attention for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2103.00853/) (2021)
- [Variational Structured Attention Networks for Deep Visual Representation Learning](https://mlsanity.com/paper/2103.03510/) (2021)
- [GenDepth: Generalizing Monocular Depth Estimation for Arbitrary Camera Parameters via Ground Plane Embedding](https://mlsanity.com/paper/2312.06021/) (2023)
- [Investigation of Hierarchical Spectral Vision Transformer Architecture for Classification of Hyperspectral Imagery](https://mlsanity.com/paper/2409.09244/) (2024)
- [Depth-Relative Self Attention for Monocular Depth Estimation](https://mlsanity.com/paper/2304.12849/) (2023)
- [FG-Depth: Flow-Guided Unsupervised Monocular Depth Estimation](https://mlsanity.com/paper/2301.08414/) (2023)

## Abstract (arXiv)

Solving depth estimation with monocular cameras enables the possibility of widespread use of cameras as low-cost depth estimation sensors in applications such as autonomous driving and robotics. However, learning such a scalable depth estimation model would require a lot of labeled data which is expensive to collect. There are two popular existing approaches which do not require annotated depth maps: (i) using labeled synthetic and unlabeled real data in an adversarial framework to predict more accurate depth, and (ii) unsupervised models which exploit geometric structure across space and time in monocular video frames. Ideally, we would like to leverage features provided by both approaches as they complement each other; however, existing methods do not adequately exploit these additive benefits. We present S3Net, a self-supervised framework which combines these complementary features: we use synthetic and real-world images for training while exploiting geometric, temporal, as well as semantic constraints. Our novel consolidated architecture provides a new state-of-the-art in self-supervised depth estimation using monocular videos. We present a unique way to train this self-supervised framework, and achieve (i) more than 15% improvement over previous synthetic supervised approaches that use domain adaptation and (ii) more than 10% improvement over previous self-supervised approaches which exploit geometric constraints from the real data.