# S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation

arXiv: https://arxiv.org/abs/2104.00877 · HTML: https://mlsanity.com/paper/2104.00877/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 60 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 30 of 48)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Unsupervised Domain Adaptation with Residual Transfer Networks](https://mlsanity.com/paper/1602.04433/) (2016)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Generalizing Across Domains via Cross-Gradient Training](https://mlsanity.com/paper/1804.10745/) (2018)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Style Normalization and Restitution for Generalizable Person Re-identification](https://mlsanity.com/paper/2005.11037/) (2020)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)

## Built on by (most cited first, 23 of 23)

- [Self-Supervised Monocular Depth Estimation with Internal Feature Fusion](https://mlsanity.com/paper/2110.09482/) (2021)
- [Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey](https://mlsanity.com/paper/2406.19675/) (2024)
- [G2-MonoDepth: A General Framework of Generalized Depth Inference from Monocular RGB+X Data](https://mlsanity.com/paper/2310.15422/) (2023)
- [Depth Completion using Geometry-Aware Embedding](https://mlsanity.com/paper/2203.10912/) (2022)
- [A review on vision-based analysis for automatic dietary assessment](https://mlsanity.com/paper/2108.02947/) (2021)
- [Mind The Edge: Refining Depth Edges in Sparsely-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2212.05315/) (2022)
- [SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2111.09692/) (2021)
- [3D-PL: Domain Adaptive Depth Estimation with 3D-aware Pseudo-Labeling](https://mlsanity.com/paper/2209.09231/) (2022)
- [BDIS: Bayesian Dense Inverse Searching Method for Real-Time Stereo Surgical Image Matching](https://mlsanity.com/paper/2205.03133/) (2022)
- [Dense Depth Distillation with Out-of-Distribution Simulated Images](https://mlsanity.com/paper/2208.12464/) (2022)
- [Bayesian dense inverse searching algorithm for real-time stereo matching in minimally invasive surgery](https://mlsanity.com/paper/2106.07136/) (2021)
- [Neural Contourlet Network for Monocular 360 Depth Estimation](https://mlsanity.com/paper/2208.01817/) (2022)
- [DeepMLE: A Robust Deep Maximum Likelihood Estimator for Two-view Structure from Motion](https://mlsanity.com/paper/2210.05517/) (2022)
- [Meta-Optimization for Higher Model Generalizability in Single-Image Depth Prediction](https://mlsanity.com/paper/2305.07269/) (2023)
- [Bridging Spectral-wise and Multi-spectral Depth Estimation via Geometry-guided Contrastive Learning](https://mlsanity.com/paper/2503.00793/) (2025)
- [FlowDepth: Decoupling Optical Flow for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2403.19294/) (2024)
- [ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation](https://mlsanity.com/paper/2407.16508/) (2024)
- [Consistency Regularisation for Unsupervised Domain Adaptation in Monocular Depth Estimation](https://mlsanity.com/paper/2405.17704/) (2024)
- [Boosting Generalizability towards Zero-Shot Cross-Dataset Single-Image Indoor Depth by Meta-Initialization](https://mlsanity.com/paper/2409.02486/) (2024)
- [Consistent Depth Prediction under Various Illuminations using Dilated Cross Attention](https://mlsanity.com/paper/2112.08006/) (2021)
- [Dual-branch Graph Feature Learning for NLOS Imaging](https://mlsanity.com/paper/2502.19683/) (2025)
- [FreeDNA: Endowing Domain Adaptation of Diffusion-Based Dense Prediction with Training-Free Domain Noise Alignment](https://mlsanity.com/paper/2506.22509/) (2025)
- [A Stereo Visual SLAM System Using Object-Level Motion Estimation and Geometric Filtering Based on Cross Disparity](https://mlsanity.com/paper/2607.02005/) (2026)

## Abstract (arXiv)

Human can infer the 3D geometry of a scene from a sketch instead of a realistic image, which indicates that the spatial structure plays a fundamental role in understanding the depth of scenes. We are the first to explore the learning of a depth-specific structural representation, which captures the essential feature for depth estimation and ignores irrelevant style information. Our S2R-DepthNet (Synthetic to Real DepthNet) can be well generalized to unseen real-world data directly even though it is only trained on synthetic data. S2R-DepthNet consists of: a) a Structure Extraction (STE) module which extracts a domaininvariant structural representation from an image by disentangling the image into domain-invariant structure and domain-specific style components, b) a Depth-specific Attention (DSA) module, which learns task-specific knowledge to suppress depth-irrelevant structures for better depth estimation and generalization, and c) a depth prediction module (DP) to predict depth from the depth-specific representation. Without access of any real-world images, our method even outperforms the state-of-the-art unsupervised domain adaptation methods which use real-world images of the target domain for training. In addition, when using a small amount of labeled real-world data, we achieve the state-ofthe-art performance under the semi-supervised setting. The code and trained models are available at https://github.com/microsoft/S2R-DepthNet.