# Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation

arXiv: https://arxiv.org/abs/2012.10782 · HTML: https://mlsanity.com/paper/2012.10782/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2020.

## Built on by (most cited first, 30 of 66)

- [DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2111.14887/) (2021)
- [MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation](https://mlsanity.com/paper/2212.01322/) (2022)
- [HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.13132/) (2022)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [OrienterNet: Visual Localization in 2D Public Maps with Neural Matching](https://mlsanity.com/paper/2304.02009/) (2023)
- [Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation](https://mlsanity.com/paper/2104.13613/) (2021)
- [TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts](https://mlsanity.com/paper/2307.15324/) (2023)
- [SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidance](https://mlsanity.com/paper/2311.16241/) (2023)
- [Neural Preset for Color Style Transfer](https://mlsanity.com/paper/2303.13511/) (2023)
- [BiFuse++: Self-supervised and Efficient Bi-projection Fusion for 360 Depth Estimation](https://mlsanity.com/paper/2209.02952/) (2022)
- [Domain Adaptive and Generalizable Network Architectures and Training Strategies for Semantic Image Segmentation](https://mlsanity.com/paper/2304.13615/) (2023)
- [Improving Semi-Supervised and Domain-Adaptive Semantic Segmentation with Self-Supervised Depth Estimation](https://mlsanity.com/paper/2108.12545/) (2021)
- [DenseMTL: Cross-task Attention Mechanism for Dense Multi-task Learning](https://mlsanity.com/paper/2206.08927/) (2022)
- [Unsupervised Semantic Segmentation Through Depth-Guided Feature Correlation and Sampling](https://mlsanity.com/paper/2309.12378/) (2023)
- [DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data](https://mlsanity.com/paper/2403.15389/) (2024)
- [Fully Self-Supervised Depth Estimation from Defocus Clue](https://mlsanity.com/paper/2303.10752/) (2023)
- [Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Survey](https://mlsanity.com/paper/2310.12904/) (2023)
- [Plugging Self-Supervised Monocular Depth into Unsupervised Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2110.06685/) (2021)
- [CLUDA : Contrastive Learning in Unsupervised Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2208.14227/) (2022)
- [SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Images](https://mlsanity.com/paper/2403.10662/) (2024)
- [MICDrop: Masking Image and Depth Features via Complementary Dropout for Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2408.16478/) (2024)
- [Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration](https://mlsanity.com/paper/2212.07592/) (2022)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [Prospective Role of Foundation Models in Advancing Autonomous Vehicles](https://mlsanity.com/paper/2405.02288/) (2023)
- [A Good Foundation is Worth Many Labels: Label-Efficient Panoptic Segmentation](https://mlsanity.com/paper/2405.19035/) (2024)
- [When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review](https://mlsanity.com/paper/2307.14382/) (2023)
- [FloCoDe: Unbiased Dynamic Scene Graph Generation with Temporal Consistency and Correlation Debiasing](https://mlsanity.com/paper/2310.16073/) (2023)
- [2D Feature Distillation for Weakly- and Semi-Supervised 3D Semantic Segmentation](https://mlsanity.com/paper/2311.15605/) (2023)
- [Proximity QA: Unleashing the Power of Multi-Modal Large Language Models for Spatial Proximity Analysis](https://mlsanity.com/paper/2401.17862/) (2024)
- [Sound and Visual Representation Learning with Multiple Pretraining Tasks](https://mlsanity.com/paper/2201.01046/) (2022)

## Abstract (arXiv)

Training deep networks for semantic segmentation requires large amounts of labeled training data, which presents a major challenge in practice, as labeling segmentation masks is a highly labor-intensive process. To address this issue, we present a framework for semi-supervised semantic segmentation, which is enhanced by self-supervised monocular depth estimation from unlabeled image sequences. In particular, we propose three key contributions: (1) We transfer knowledge from features learned during self-supervised depth estimation to semantic segmentation, (2) we implement a strong data augmentation by blending images and labels using the geometry of the scene, and (3) we utilize the depth feature diversity as well as the level of difficulty of learning depth in a student-teacher framework to select the most useful samples to be annotated for semantic segmentation. We validate the proposed model on the Cityscapes dataset, where all three modules demonstrate significant performance gains, and we achieve state-of-the-art results for semi-supervised semantic segmentation. The implementation is available at https://github.com/lhoyer/improving_segmentation_with_selfsupervised_depth.