# X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation

arXiv: https://arxiv.org/abs/2110.12516 · HTML: https://mlsanity.com/paper/2110.12516/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 24 Oct 2021.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 30 of 34)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras](https://mlsanity.com/paper/1904.04998/) (2019)
- [Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos](https://mlsanity.com/paper/1811.06152/) (2018)
- [Learning Depth from Monocular Videos using Direct Methods](https://mlsanity.com/paper/1712.00175/) (2017)
- [Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance](https://mlsanity.com/paper/2007.06936/) (2020)
- [Feature-metric Loss for Self-supervised Learning of Depth and Egomotion](https://mlsanity.com/paper/2007.10603/) (2020)
- [Semantically-Guided Representation Learning for Self-Supervised Monocular Depth](https://mlsanity.com/paper/2002.12319/) (2020)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volume](https://mlsanity.com/paper/2003.13951/) (2020)
- [HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2012.07356/) (2020)
- [Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding](https://mlsanity.com/paper/1810.06125/) (2018)
- [SegStereo: Exploiting Semantic Information for Disparity Estimation](https://mlsanity.com/paper/1807.11699/) (2018)
- [The Edge of Depth: Explicit Constraints between Segmentation and Depth](https://mlsanity.com/paper/2004.00171/) (2020)
- [Don't Forget The Past: Recurrent Depth Estimation from Monocular Video](https://mlsanity.com/paper/2001.02613/) (2020)
- [InverseForm: A Loss Function for Structured Boundary-Aware Segmentation](https://mlsanity.com/paper/2104.02745/) (2021)
- [DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling](https://mlsanity.com/paper/2001.00987/) (2019)
- [SynDistNet: Self-Supervised Monocular Fisheye Camera Distance Estimation Synergized with Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2008.04017/) (2020)
- [Distilled Semantics for Comprehensive Scene Understanding from Videos](https://mlsanity.com/paper/2003.14030/) (2020)
- [Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces](https://mlsanity.com/paper/2012.08859/) (2020)
- [Learning Scene Structure Guidance via Cross-Task Knowledge Transfer for Single Depth Super-Resolution](https://mlsanity.com/paper/2103.12955/) (2021)

## Built on by (most cited first, 12 of 12)

- [X3KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detection](https://mlsanity.com/paper/2303.02203/) (2023)
- [MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimation](https://mlsanity.com/paper/2307.14336/) (2023)
- [EGA-Depth: Efficient Guided Attention for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2304.03369/) (2023)
- [DistractFlow: Improving Optical Flow Estimation via Realistic Distractions and Pseudo-Labeling](https://mlsanity.com/paper/2303.14078/) (2023)
- [FutureDepth: Learning to Predict the Future Improves Video Depth Estimation](https://mlsanity.com/paper/2403.12953/) (2024)
- [Panoptic, Instance and Semantic Relations: A Relational Context Encoder to Enhance Panoptic Segmentation](https://mlsanity.com/paper/2204.05370/) (2022)
- [DejaVu: Conditional Regenerative Learning to Enhance Dense Prediction](https://mlsanity.com/paper/2303.01573/) (2023)
- [Self-supervised Hypergraphs for Learning Multiple World Interpretations](https://mlsanity.com/paper/2308.07615/) (2023)
- [SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2111.09692/) (2021)
- [ROIFormer: Semantic-Aware Region of Interest Transformer for Efficient Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2212.05729/) (2022)
- [SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations](https://mlsanity.com/paper/2404.08135/) (2024)
- [STATIC : Surface Temporal Affine for TIme Consistency in Video Monocular Depth Estimation](https://mlsanity.com/paper/2412.01090/) (2024)

## Abstract (arXiv)

In this paper, we propose a novel method, X-Distill, to improve the self-supervised training of monocular depth via cross-task knowledge distillation from semantic segmentation to depth estimation. More specifically, during training, we utilize a pretrained semantic segmentation teacher network and transfer its semantic knowledge to the depth network. In order to enable such knowledge distillation across two different visual tasks, we introduce a small, trainable network that translates the predicted depth map to a semantic segmentation map, which can then be supervised by the teacher network. In this way, this small network enables the backpropagation from the semantic segmentation teacher's supervision to the depth network during training. In addition, since the commonly used object classes in semantic segmentation are not directly transferable to depth, we study the visual and geometric characteristics of the objects and design a new way of grouping them that can be shared by both tasks. It is noteworthy that our approach only modifies the training process and does not incur additional computation during inference. We extensively evaluate the efficacy of our proposed approach on the standard KITTI benchmark and compare it with the latest state of the art. We further test the generalizability of our approach on Make3D. Overall, the results show that our approach significantly improves the depth estimation accuracy and outperforms the state of the art.