# Real-World Depth Recovery via Structure Uncertainty Modeling and Inaccurate GT Depth Fitting

arXiv: https://arxiv.org/abs/2504.11820 · HTML: https://mlsanity.com/paper/2504.11820/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Builds on (most cited first, 23 of 23)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [Evaluation of CNN-based Single-Image Depth Estimation Methods](https://mlsanity.com/paper/1805.01328/) (2018)
- [Discrete Cosine Transform Network for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2104.06977/) (2021)
- [Towards Fast and Accurate Real-World Depth Super-Resolution: Benchmark Dataset and Baseline](https://mlsanity.com/paper/2104.06174/) (2021)
- [Deformable Kernel Networks for Joint Image Filtering](https://mlsanity.com/paper/1910.08373/) (2019)
- [Guided Depth Super-Resolution by Deep Anisotropic Diffusion](https://mlsanity.com/paper/2211.11592/) (2022)
- [Deep Depth Completion from Extremely Sparse Data: A Survey](https://mlsanity.com/paper/2205.05335/) (2022)
- [SGNet: Structure Guided Network via Gradient-Frequency Awareness for Depth Map Super-Resolution](https://mlsanity.com/paper/2312.05799/) (2023)
- [Deep Attentional Guided Image Filtering](https://mlsanity.com/paper/2112.06401/) (2021)
- [A Generalized Framework for Edge-preserving and Structure-preserving Image Smoothing](https://mlsanity.com/paper/1907.09642/) (2019)
- [G2-MonoDepth: A General Framework of Generalized Depth Inference from Monocular RGB+X Data](https://mlsanity.com/paper/2310.15422/) (2023)
- [Gradient-Semantic Compensation for Incremental Semantic Segmentation](https://mlsanity.com/paper/2307.10822/) (2023)
- [DC-MRTA: Decentralized Multi-Robot Task Allocation and Navigation in Complex Environments](https://mlsanity.com/paper/2209.02865/) (2022)
- [M2RNet: Multi-modal and Multi-scale Refined Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2109.07922/) (2021)

## Abstract (arXiv)

The low-quality structure in raw depth maps is prevalent in real-world RGB-D datasets, which makes real-world depth recovery a critical task in recent years. However, the lack of paired raw-ground truth (raw-GT) data in the real world poses challenges for generalized depth recovery. Existing methods insufficiently consider the diversity of structure misalignment in raw depth maps, which leads to poor generalization in real-world depth recovery. Notably, random structure misalignments are not limited to raw depth data but also affect GT depth in real-world datasets. In the proposed method, we tackle the generalization problem from both input and output perspectives. For input, we enrich the diversity of structure misalignment in raw depth maps by designing a new raw depth generation pipeline, which helps the network avoid overfitting to a specific condition. Furthermore, a structure uncertainty module is designed to explicitly identify the misaligned structure for input raw depth maps to better generalize in unseen scenarios. Notably the well-trained depth foundation model (DFM) can help the structure uncertainty module estimate the structure uncertainty better. For output, a robust feature alignment module is designed to precisely align with the accurate structure of RGB images avoiding the interference of inaccurate GT depth. Extensive experiments on multiple datasets demonstrate the proposed method achieves competitive accuracy and generalization capabilities across various challenging raw depth maps.