# T2Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks

arXiv: https://arxiv.org/abs/1808.01454 · HTML: https://mlsanity.com/paper/1808.01454/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2018.

## Built on by (most cited first, 30 of 70)

- [Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging](https://mlsanity.com/paper/2105.14021/) (2021)
- [SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2204.03636/) (2022)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [Regularizing Nighttime Weirdness: Efficient Self-supervised Monocular Depth Estimation in the Dark](https://mlsanity.com/paper/2108.03830/) (2021)
- [PanoDiffusion: 360-degree Panorama Outpainting via Diffusion](https://mlsanity.com/paper/2307.03177/) (2023)
- [HairStep: Transfer Synthetic to Real Using Strand and Depth Maps for Single-View 3D Hair Modeling](https://mlsanity.com/paper/2303.02700/) (2023)
- [Single Image Depth Estimation: An Overview](https://mlsanity.com/paper/2104.06456/) (2021)
- [Domain Adaptation for Underwater Image Enhancement](https://mlsanity.com/paper/2108.09650/) (2021)
- [CrossLoc: Scalable Aerial Localization Assisted by Multimodal Synthetic Data](https://mlsanity.com/paper/2112.09081/) (2021)
- [Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction](https://mlsanity.com/paper/2303.09792/) (2023)
- [Monocular Depth Estimation through Virtual-world Supervision and Real-world SfM Self-Supervision](https://mlsanity.com/paper/2103.12209/) (2021)
- [S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation](https://mlsanity.com/paper/2104.00877/) (2021)
- [SyreaNet: A Physically Guided Underwater Image Enhancement Framework Integrating Synthetic and Real Images](https://mlsanity.com/paper/2302.08269/) (2023)
- [Outdoor inverse rendering from a single image using multiview self-supervision](https://mlsanity.com/paper/2102.06591/) (2021)
- [PatchRefiner: Leveraging Synthetic Data for Real-Domain High-Resolution Monocular Metric Depth Estimation](https://mlsanity.com/paper/2406.06679/) (2024)
- [Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey](https://mlsanity.com/paper/2406.19675/) (2024)
- [PanopticDepth: A Unified Framework for Depth-aware Panoptic Segmentation](https://mlsanity.com/paper/2206.00468/) (2022)
- [IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation](https://mlsanity.com/paper/2407.10937/) (2024)
- [Self-Supervised Learning of Domain Invariant Features for Depth Estimation](https://mlsanity.com/paper/2106.02594/) (2021)
- [Learning Feature Decomposition for Domain Adaptive Monocular Depth Estimation](https://mlsanity.com/paper/2208.00160/) (2022)
- [SynRS3D: A Synthetic Dataset for Global 3D Semantic Understanding from Monocular Remote Sensing Imagery](https://mlsanity.com/paper/2406.18151/) (2024)
- [Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration](https://mlsanity.com/paper/2406.18516/) (2024)
- [Estimating Image Depth in the Comics Domain](https://mlsanity.com/paper/2110.03575/) (2021)
- [Towards Scale Consistent Monocular Visual Odometry by Learning from the Virtual World](https://mlsanity.com/paper/2203.05712/) (2022)
- [Unsupervised Spike Depth Estimation via Cross-modality Cross-domain Knowledge Transfer](https://mlsanity.com/paper/2208.12527/) (2022)
- [AdaStereo: An Efficient Domain-Adaptive Stereo Matching Approach](https://mlsanity.com/paper/2112.04974/) (2021)
- [deepNIR: Datasets for generating synthetic NIR images and improved fruit detection system using deep learning techniques](https://mlsanity.com/paper/2203.09091/) (2022)
- [3D-PL: Domain Adaptive Depth Estimation with 3D-aware Pseudo-Labeling](https://mlsanity.com/paper/2209.09231/) (2022)
- [Layered Depth Refinement with Mask Guidance](https://mlsanity.com/paper/2206.03048/) (2022)
- [Task-guided Domain Gap Reduction for Monocular Depth Prediction in Endoscopy](https://mlsanity.com/paper/2310.01663/) (2023)

## Abstract (arXiv)

Current methods for single-image depth estimation use training datasets with real image-depth pairs or stereo pairs, which are not easy to acquire. We propose a framework, trained on synthetic image-depth pairs and unpaired real images, that comprises an image translation network for enhancing realism of input images, followed by a depth prediction network. A key idea is having the first network act as a wide-spectrum input translator, taking in either synthetic or real images, and ideally producing minimally modified realistic images. This is done via a reconstruction loss when the training input is real, and GAN loss when synthetic, removing the need for heuristic self-regularization. The second network is trained on a task loss for synthetic image-depth pairs, with extra GAN loss to unify real and synthetic feature distributions. Importantly, the framework can be trained end-to-end, leading to good results, even surpassing early deep-learning methods that use real paired data.