# Deep Surface Normal Estimation with Hierarchical RGB-D Fusion

arXiv: https://arxiv.org/abs/1904.03405 · HTML: https://mlsanity.com/paper/1904.03405/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2019.

## Built on by (most cited first, 20 of 20)

- [Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods](https://mlsanity.com/paper/2110.14446/) (2021)
- [TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network](https://mlsanity.com/paper/2108.03990/) (2021)
- [Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion](https://mlsanity.com/paper/2108.05009/) (2021)
- [Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction](https://mlsanity.com/paper/2112.02252/) (2021)
- [Structured Attention Composition for Temporal Action Localization](https://mlsanity.com/paper/2205.09956/) (2022)
- [Egocentric Scene Understanding via Multimodal Spatial Rectifier](https://mlsanity.com/paper/2207.07077/) (2022)
- [Self-Supervised Ego-Motion Estimation Based on Multi-Layer Fusion of RGB and Inferred Depth](https://mlsanity.com/paper/2203.01557/) (2022)
- [Modeling 3D Surface Manifolds with a Locally Conditioned Atlas](https://mlsanity.com/paper/2102.05984/) (2021)
- [DCL: Differential Contrastive Learning for Geometry-Aware Depth Synthesis](https://mlsanity.com/paper/2107.13087/) (2021)
- [BadPart: Unified Black-box Adversarial Patch Attacks against Pixel-wise Regression Tasks](https://mlsanity.com/paper/2404.00924/) (2024)
- [Normal Transformer: Extracting Surface Geometry from LiDAR Points Enhanced by Visual Semantics](https://mlsanity.com/paper/2211.10580/) (2022)
- [Exchanging-based Multimodal Fusion with Transformer](https://mlsanity.com/paper/2309.02190/) (2023)
- [Automatic Detection of Alzheimer's Disease with Multi-Modal Fusion of Clinical MRI Scans](https://mlsanity.com/paper/2311.18245/) (2023)
- [How Intermodal Interaction Affects the Performance of Deep Multimodal Fusion for Mixed-Type Time Series](https://mlsanity.com/paper/2406.15098/) (2024)
- [Countering Multi-modal Representation Collapse through Rank-targeted Fusion](https://mlsanity.com/paper/2511.06450/) (2025)
- [Stabilizing Multimodal Autoencoders: A Theoretical and Empirical Analysis of Fusion Strategies](https://mlsanity.com/paper/2512.20749/) (2025)
- [RXFOOD: Plug-in RGB-X Fusion for Object of Interest Detection](https://mlsanity.com/paper/2306.12621/) (2023)
- [Dissecting RGB-D Learning for Improved Multi-modal Fusion](https://mlsanity.com/paper/2308.10019/) (2023)
- [Towards Robust Time-of-Flight Depth Denoising with Confidence-Aware Diffusion Model](https://mlsanity.com/paper/2503.19448/) (2025)
- [M3Depth: Wavelet-Enhanced Depth Estimation on Mars via Mutual Boosting of Dual-Modal Data](https://mlsanity.com/paper/2505.14159/) (2025)

## Abstract (arXiv)

The growing availability of commodity RGB-D cameras has boosted the applications in the field of scene understanding. However, as a fundamental scene understanding task, surface normal estimation from RGB-D data lacks thorough investigation. In this paper, a hierarchical fusion network with adaptive feature re-weighting is proposed for surface normal estimation from a single RGB-D image. Specifically, the features from color image and depth are successively integrated at multiple scales to ensure global surface smoothness while preserving visually salient details. Meanwhile, the depth features are re-weighted with a confidence map estimated from depth before merging into the color branch to avoid artifacts caused by input depth corruption. Additionally, a hybrid multi-scale loss function is designed to learn accurate normal estimation given noisy ground-truth dataset. Extensive experimental results validate the effectiveness of the fusion strategy and the loss design, outperforming state-of-the-art normal estimation schemes.