# High-resolution Depth Maps Imaging via Attention-based Hierarchical Multi-modal Fusion

arXiv: https://arxiv.org/abs/2104.01530 · HTML: https://mlsanity.com/paper/2104.01530/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 70 references that are in the map.
- First posted on arXiv 4 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [Pixel-Adaptive Convolutional Neural Networks](https://mlsanity.com/paper/1904.05373/) (2019)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion](https://mlsanity.com/paper/1910.04066/) (2019)
- [Deformable Kernel Networks for Joint Image Filtering](https://mlsanity.com/paper/1910.08373/) (2019)
- [Joint Image Filtering with Deep Convolutional Networks](https://mlsanity.com/paper/1710.04200/) (2017)
- [Fast End-to-End Trainable Guided Filter](https://mlsanity.com/paper/1803.05619/) (2018)
- [Learning Scene Structure Guidance via Cross-Task Knowledge Transfer for Single Depth Super-Resolution](https://mlsanity.com/paper/2103.12955/) (2021)
- [ATGV-Net: Accurate Depth Super-Resolution](https://mlsanity.com/paper/1607.07988/) (2016)
- [Pyramid Attention Networks for Image Restoration](https://mlsanity.com/paper/2004.13824/) (2020)
- [Robust super-resolution depth imaging via a multi-feature fusion deep network](https://mlsanity.com/paper/2011.11444/) (2020)
- [Long-range depth imaging using a single-photon detector array and non-local data fusion](https://mlsanity.com/paper/1812.04943/) (2018)

## Built on by (most cited first, 20 of 20)

- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [SGNet: Structure Guided Network via Gradient-Frequency Awareness for Depth Map Super-Resolution](https://mlsanity.com/paper/2312.05799/) (2023)
- [Guided Depth Map Super-resolution: A Survey](https://mlsanity.com/paper/2302.09598/) (2023)
- [Recurrent Structure Attention Guidance for Depth Super-Resolution](https://mlsanity.com/paper/2301.13419/) (2023)
- [DORNet: A Degradation Oriented and Regularized Network for Blind Depth Super-Resolution](https://mlsanity.com/paper/2410.11666/) (2024)
- [RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2405.10357/) (2024)
- [Learning Continuous Depth Representation via Geometric Spatial Aggregator](https://mlsanity.com/paper/2212.03499/) (2022)
- [Scene Prior Filtering for Depth Super-Resolution](https://mlsanity.com/paper/2402.13876/) (2024)
- [SpatioTemporal Difference Network for Video Depth Super-Resolution](https://mlsanity.com/paper/2508.01259/) (2025)
- [DSR-Diff: Depth Map Super-Resolution with Diffusion Model](https://mlsanity.com/paper/2311.09919/) (2023)
- [MoCoLSK: Modality Conditioned High-Resolution Downscaling for Land Surface Temperature](https://mlsanity.com/paper/2409.19835/) (2024)
- [Multi-Order Matching Network for Alignment-Free Depth Super-Resolution](https://mlsanity.com/paper/2511.16361/) (2025)
- [NAIMA: Semantics Aware RGB Guided Depth Super-Resolution](https://mlsanity.com/paper/2604.04407/) (2026)
- [Cutting-Edge Techniques for Depth Map Super-Resolution](https://mlsanity.com/paper/2306.15244/) (2023)
- [Supersampling of Data from Structured-light Scanner with Deep Learning](https://mlsanity.com/paper/2311.07432/) (2023)
- [Guided Image Restoration via Simultaneous Feature and Image Guided Fusion](https://mlsanity.com/paper/2312.08853/) (2023)
- [IGAF: Incremental Guided Attention Fusion for Depth Super-Resolution](https://mlsanity.com/paper/2501.01723/) (2025)
- [DegBins: Degradation-Driven Binning for Depth Super-Resolution](https://mlsanity.com/paper/2605.09628/) (2026)
- [WAVE: Reversing the Guidance Hierarchy for Coarse-to-Fine Guided Depth Super-Resolution](https://mlsanity.com/paper/2608.25302/) (2026)
- [Passive LWIR Hyperspectral Ranging via Transmittance Extraction and Distance Alignment](https://mlsanity.com/paper/2609.28967/) (2026)

## Abstract (arXiv)

Depth map records distance between the viewpoint and objects in the scene, which plays a critical role in many real-world applications. However, depth map captured by consumer-grade RGB-D cameras suffers from low spatial resolution. Guided depth map super-resolution (DSR) is a popular approach to address this problem, which attempts to restore a high-resolution (HR) depth map from the input low-resolution (LR) depth and its coupled HR RGB image that serves as the guidance. The most challenging problems for guided DSR are how to correctly select consistent structures and propagate them, and properly handle inconsistent ones. In this paper, we propose a novel attention-based hierarchical multi-modal fusion (AHMF) network for guided DSR. Specifically, to effectively extract and combine relevant information from LR depth and HR guidance, we propose a multi-modal attention based fusion (MMAF) strategy for hierarchical convolutional layers, including a feature enhance block to select valuable features and a feature recalibration block to unify the similarity metrics of modalities with different appearance characteristics. Furthermore, we propose a bi-directional hierarchical feature collaboration (BHFC) module to fully leverage low-level spatial information and high-level structure information among multi-scale features. Experimental results show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.