# Transformer-based Network for RGB-D Saliency Detection

arXiv: https://arxiv.org/abs/2112.00582 · HTML: https://mlsanity.com/paper/2112.00582/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 1 Dec 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Structure-measure: A New Way to Evaluate Foreground Maps](https://mlsanity.com/paper/1708.00786/) (2017)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Enhanced-alignment Measure for Binary Foreground Map Evaluation](https://mlsanity.com/paper/1805.10421/) (2018)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks](https://mlsanity.com/paper/1907.06781/) (2019)
- [Salient Object Detection: A Benchmark](https://mlsanity.com/paper/1501.02741/) (2015)
- [Feature Pyramid Transformer](https://mlsanity.com/paper/2007.09451/) (2020)
- [Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2007.06227/) (2020)
- [A Single Stream Network for Robust and Real-time RGB-D Salient Object Detection](https://mlsanity.com/paper/2007.06811/) (2020)
- [RGB-D Salient Object Detection with Cross-Modality Modulation and Selection](https://mlsanity.com/paper/2007.07051/) (2020)
- [Cross-Modal Weighting Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2007.04901/) (2020)
- [RGBD Salient Object Detection via Deep Fusion](https://mlsanity.com/paper/1607.03333/) (2016)
- [Progressively Guided Alternate Refinement Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2008.07064/) (2020)
- [Saliency Detection for Stereoscopic Images Based on Depth Confidence Analysis and Multiple Cues Fusion](https://mlsanity.com/paper/1710.05174/) (2017)

## Built on by (most cited first, 2 of 2)

- [HRTransNet: HRFormer-Driven Two-Modality Salient Object Detection](https://mlsanity.com/paper/2301.03036/) (2023)
- [Unleashing the Power of CNN and Transformer for Balanced RGB-Event Video Recognition](https://mlsanity.com/paper/2312.11128/) (2023)

## Abstract (arXiv)

RGB-D saliency detection integrates information from both RGB images and depth maps to improve prediction of salient regions under challenging conditions. The key to RGB-D saliency detection is to fully mine and fuse information at multiple scales across the two modalities. Previous approaches tend to apply the multi-scale and multi-modal fusion separately via local operations, which fails to capture long-range dependencies. Here we propose a transformer-based network to address this issue. Our proposed architecture is composed of two modules: a transformer-based within-modality feature enhancement module (TWFEM) and a transformer-based feature fusion module (TFFM). TFFM conducts a sufficient feature fusion by integrating features from multiple scales and two modalities over all positions simultaneously. TWFEM enhances feature on each scale by selecting and integrating complementary information from other scales within the same modality before TFFM. We show that transformer is a uniform operation which presents great efficacy in both feature fusion and feature enhancement, and simplifies the model design. Extensive experimental results on six benchmark datasets demonstrate that our proposed network performs favorably against state-of-the-art RGB-D saliency detection methods.