# Learning Scene Structure Guidance via Cross-Task Knowledge Transfer for Single Depth Super-Resolution

arXiv: https://arxiv.org/abs/2103.12955 · HTML: https://mlsanity.com/paper/2103.12955/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [Deep Back-Projection Networks For Super-Resolution](https://mlsanity.com/paper/1803.02735/) (2018)
- [PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing](https://mlsanity.com/paper/1805.04409/) (2018)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [Demystifying Neural Style Transfer](https://mlsanity.com/paper/1701.01036/) (2017)
- [Pixel-Adaptive Convolutional Neural Networks](https://mlsanity.com/paper/1904.05373/) (2019)
- [Structure-Preserving Super Resolution with Gradient Guidance](https://mlsanity.com/paper/2003.13081/) (2020)
- [Modality Distillation with Multiple Stream Networks for Action Recognition](https://mlsanity.com/paper/1806.07110/) (2018)
- [Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation](https://mlsanity.com/paper/1803.11029/) (2018)
- [Learning Monocular Depth by Distilling Cross-domain Stereo Networks](https://mlsanity.com/paper/1808.06586/) (2018)
- [AdaDepth: Unsupervised Content Congruent Adaptation for Depth Estimation](https://mlsanity.com/paper/1803.01599/) (2018)
- [Joint Image Filtering with Deep Convolutional Networks](https://mlsanity.com/paper/1710.04200/) (2017)
- [Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Prediction](https://mlsanity.com/paper/1903.07309/) (2019)
- [Knowledge Transfer via Dense Cross-Layer Mutual-Distillation](https://mlsanity.com/paper/2008.07816/) (2020)
- [Guided Image-to-Image Translation with Bi-Directional Feature Transformation](https://mlsanity.com/paper/1910.11328/) (2019)
- [UM-Adapt: Unsupervised Multi-Task Adaptation Using Adversarial Cross-Task Distillation](https://mlsanity.com/paper/1908.03884/) (2019)
- [ATGV-Net: Accurate Depth Super-Resolution](https://mlsanity.com/paper/1607.07988/) (2016)
- [Deformable kernel networks for guided depth map upsampling](https://mlsanity.com/paper/1903.11286/) (2019)
- [Deeply Supervised Depth Map Super-Resolution as Novel View Synthesis](https://mlsanity.com/paper/1808.08688/) (2018)
- [Combining Deep and Depth: Deep Learning and Face Depth Maps for Driver Attention Monitoring](https://mlsanity.com/paper/1812.05831/) (2018)

## Built on by (most cited first, 30 of 30)

- [Discrete Cosine Transform Network for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2104.06977/) (2021)
- [BridgeNet: A Joint Learning Network of Depth Map Super-Resolution and Monocular Depth Estimation](https://mlsanity.com/paper/2107.12541/) (2021)
- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [Learning Graph Regularisation for Guided Super-Resolution](https://mlsanity.com/paper/2203.14297/) (2022)
- [SGNet: Structure Guided Network via Gradient-Frequency Awareness for Depth Map Super-Resolution](https://mlsanity.com/paper/2312.05799/) (2023)
- [Memory-augmented Deep Unfolding Network for Guided Image Super-resolution](https://mlsanity.com/paper/2203.04960/) (2022)
- [Guided Depth Map Super-resolution: A Survey](https://mlsanity.com/paper/2302.09598/) (2023)
- [High-resolution Depth Maps Imaging via Attention-based Hierarchical Multi-modal Fusion](https://mlsanity.com/paper/2104.01530/) (2021)
- [Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution](https://mlsanity.com/paper/2306.00386/) (2023)
- [Recurrent Structure Attention Guidance for Depth Super-Resolution](https://mlsanity.com/paper/2301.13419/) (2023)
- [Structure Flow-Guided Network for Real Depth Super-Resolution](https://mlsanity.com/paper/2301.13416/) (2023)
- [DORNet: A Degradation Oriented and Regularized Network for Blind Depth Super-Resolution](https://mlsanity.com/paper/2410.11666/) (2024)
- [X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation](https://mlsanity.com/paper/2110.12516/) (2021)
- [DuCos: Duality Constrained Depth Super-Resolution via Foundation Model](https://mlsanity.com/paper/2503.04171/) (2025)
- [RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2405.10357/) (2024)
- [Decoupling Fine Detail and Global Geometry for Compressed Depth Map Super-Resolution](https://mlsanity.com/paper/2411.03239/) (2024)
- [Scene Prior Filtering for Depth Super-Resolution](https://mlsanity.com/paper/2402.13876/) (2024)
- [U-Know-DiffPAN: An Uncertainty-aware Knowledge Distillation Diffusion Framework with Details Enhancement for PAN-Sharpening](https://mlsanity.com/paper/2412.06243/) (2024)
- [SpatioTemporal Difference Network for Video Depth Super-Resolution](https://mlsanity.com/paper/2508.01259/) (2025)
- [Weakly-Supervised Monocular Depth Estimationwith Resolution-Mismatched Data](https://mlsanity.com/paper/2109.11573/) (2021)
- [Harmonizing Light and Darkness: A Symphony of Prior-guided Data Synthesis and Adaptive Focus for Nighttime Flare Removal](https://mlsanity.com/paper/2404.00313/) (2024)
- [Learning Hierarchical Color Guidance for Depth Map Super-Resolution](https://mlsanity.com/paper/2403.07290/) (2024)
- [Multi-Order Matching Network for Alignment-Free Depth Super-Resolution](https://mlsanity.com/paper/2511.16361/) (2025)
- [360^circ High-Resolution Depth Estimation via Uncertainty-aware Structural Knowledge Transfer](https://mlsanity.com/paper/2304.07967/) (2023)
- [DeepLight: Reconstructing High-Resolution Observations of Nighttime Light With Multi-Modal Remote Sensing Data](https://mlsanity.com/paper/2402.15659/) (2024)
- [IGAF: Incremental Guided Attention Fusion for Depth Super-Resolution](https://mlsanity.com/paper/2501.01723/) (2025)
- [Guided Depth Map Super-Resolution via Multi-Scale Fusion U-shaped Mamba Network](https://mlsanity.com/paper/2508.00248/) (2025)
- [Breaking Spatial Uniformity: Prior-Guided Mamba with Radial Serialization for Lens Flare Removal](https://mlsanity.com/paper/2605.07650/) (2026)
- [DegBins: Degradation-Driven Binning for Depth Super-Resolution](https://mlsanity.com/paper/2605.09628/) (2026)
- [FoundDSR: A Generalizable Foundation Model with Guided 2D Gaussian Splatting for Depth Super-Resolution](https://mlsanity.com/paper/2609.32323/) (2026)

## Abstract (arXiv)

Existing color-guided depth super-resolution (DSR) approaches require paired RGB-D data as training samples where the RGB image is used as structural guidance to recover the degraded depth map due to their geometrical similarity. However, the paired data may be limited or expensive to be collected in actual testing environment. Therefore, we explore for the first time to learn the cross-modality knowledge at training stage, where both RGB and depth modalities are available, but test on the target dataset, where only single depth modality exists. Our key idea is to distill the knowledge of scene structural guidance from RGB modality to the single DSR task without changing its network architecture. Specifically, we construct an auxiliary depth estimation (DE) task that takes an RGB image as input to estimate a depth map, and train both DSR task and DE task collaboratively to boost the performance of DSR. Upon this, a cross-task interaction module is proposed to realize bilateral cross task knowledge transfer. First, we design a cross-task distillation scheme that encourages DSR and DE networks to learn from each other in a teacher-student role-exchanging fashion. Then, we advance a structure prediction (SP) task that provides extra structure regularization to help both DSR and DE networks learn more informative structure representations for depth recovery. Extensive experiments demonstrate that our scheme achieves superior performance in comparison with other DSR methods.