mlsanityData to 7 Oct 2026

Hierarchical Cross-modal Transformer for RGB-D Salient Object Detection

arXiv 2302.08052 ↗First posted 16 Feb 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Most of existing RGB-D salient object detection (SOD) methods follow the CNN-based paradigm, which is unable to model long-range dependencies across space and modalities due to the natural locality of CNNs. Here we propose the Hierarchical Cross-modal Transformer (HCT), a new multi-modal transformer, to tackle this problem. Unlike previous multi-modal transformers that directly connecting all patches from two modalities, we explore the cross-modal complementarity hierarchically to respect the modality gap and spatial discrepancy in unaligned regions. Specifically, we propose to use intra-modal self-attention to explore complementary global contexts, and measure spatial-aligned inter-modal attention locally to capture cross-modal correlations. In addition, we present a Feature Pyramid module for Transformer (FPT) to boost informative cross-scale integration as well as a consistency-complementarity module to disentangle the multi-modal integration path and improve the fusion adaptivity. Comprehensive experiments on a large variety of public datasets verify the efficacy of our designs and the consistent improvement over state-of-the-art models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

31 of 53 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  10. 2018Panoptic Segmentationcited by 796
  11. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
19 more
  1. 2017Structure-measure: A New Way to Evaluate Foreground Mapscited by 480
  2. 2018Enhanced-alignment Measure for Binary Foreground Map Evaluationcited by 370
  3. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  4. 2021Visual Saliency Transformercited by 175
  5. 2019Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarkscited by 133
  6. 2019Deep Learning in Video Multi-Object Tracking: A Surveycited by 107
  7. 2019A Survey of Deep Learning-based Object Detectioncited by 106
  8. 2020Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detectioncited by 72
  9. 2021Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusioncited by 56
  10. 2020Accurate RGB-D Salient Object Detection via Collaborative Learningcited by 52
  11. 2021RGB-D Saliency Detection via Cascaded Mutual Information Minimizationcited by 50
  12. 2020Cross-Modal Weighting Network for RGB-D Salient Object Detectioncited by 47
  13. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  14. 2016RGBD Salient Object Detection via Deep Fusioncited by 46
  15. 2020Cascade Graph Neural Networks for RGB-D Salient Object Detectioncited by 36
  16. 2017Saliency Detection for Stereoscopic Images Based on Depth Confidence Analysis and Multiple Cues Fusioncited by 28
  17. 2021Self-Supervised Pretraining for RGB-D Salient Object Detectioncited by 20
  18. 2019CNN-based RGB-D Salient Object Detection: Learn, Select and Fusecited by 9
  19. 2017Learning RGB-D Salient Object Detection using background enclosure, depth contrast, and top-down featurescited by 7

Built on by

Not cited yet.