# Adapting Segment Anything Model to Multi-modal Salient Object Detection with Semantic Feature Fusion Guidance

arXiv: https://arxiv.org/abs/2408.15063 · HTML: https://mlsanity.com/paper/2408.15063/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 27 Aug 2024.

## Builds on (most cited first, 21 of 21)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks](https://mlsanity.com/paper/1907.06781/) (2019)
- [Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively](https://mlsanity.com/paper/2401.02955/) (2024)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [RGB-D Salient Object Detection: A Survey](https://mlsanity.com/paper/2008.00230/) (2020)
- [Specificity-preserving RGB-D Saliency Detection](https://mlsanity.com/paper/2108.08162/) (2021)
- [RGBT Salient Object Detection: A Large-scale Dataset and Benchmark](https://mlsanity.com/paper/2007.03262/) (2020)
- [Siamese Network for RGB-D Salient Object Detection and Beyond](https://mlsanity.com/paper/2008.12134/) (2020)
- [RGB-T Image Saliency Detection via Collaborative Graph Learning](https://mlsanity.com/paper/1905.06741/) (2019)
- [VSCode: General Visual Salient and Camouflaged Object Detection with 2D Prompt Learning](https://mlsanity.com/paper/2311.15011/) (2023)
- [Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion](https://mlsanity.com/paper/2103.11832/) (2021)
- [RGBD Salient Object Detection via Deep Fusion](https://mlsanity.com/paper/1607.03333/) (2016)
- [Multi-interactive Dual-decoder for RGB-thermal Salient Object Detection](https://mlsanity.com/paper/2005.02315/) (2020)
- [ExFuse: Enhancing Feature Fusion for Semantic Segmentation](https://mlsanity.com/paper/1804.03821/) (2018)
- [UV-SAM: Adapting Segment Anything Model for Urban Village Identification](https://mlsanity.com/paper/2401.08083/) (2024)
- [MobileSal: Extremely Efficient RGB-D Salient Object Detection](https://mlsanity.com/paper/2012.13095/) (2020)
- [Saliency for Fine-grained Object Recognition in Domains with Scarce Training Data](https://mlsanity.com/paper/1808.00262/) (2018)
- [Boosting Segment Anything Model Towards Open-Vocabulary Learning](https://mlsanity.com/paper/2312.03628/) (2023)

## Built on by (most cited first, 6 of 6)

- [Evaluating SAM2 for Video Semantic Segmentation](https://mlsanity.com/paper/2512.01774/) (2025)
- [SAMSOD: Rethinking SAM Optimization for RGB-T Salient Object Detection](https://mlsanity.com/paper/2510.03689/) (2025)
- [Boosting Segment Anything Model to Generalize Visually Non-Salient Scenarios](https://mlsanity.com/paper/2601.00537/) (2026)
- [Multimodal SAM-adapter for Semantic Segmentation](https://mlsanity.com/paper/2509.10408/) (2025)
- [Adapting SAM with Dynamic Similarity Graphs for Few-Shot Parameter-Efficient Small Dense Object Detection: A Case Study of Chickpea Pods in Field Conditions](https://mlsanity.com/paper/2509.25805/) (2025)
- [RDVSv2: A Large-scale Benchmark for RGB-D Video Salient Object Detection](https://mlsanity.com/paper/2607.25392/) (2026)

## Abstract (arXiv)

Although most existing multi-modal salient object detection (SOD) methods demonstrate effectiveness through training models from scratch, the limited multi-modal data hinders these methods from reaching optimality. In this paper, we propose a novel framework to explore and exploit the powerful feature representation and zero-shot generalization ability of the pre-trained Segment Anything Model (SAM) for multi-modal SOD. Despite serving as a recent vision fundamental model, driving the class-agnostic SAM to comprehend and detect salient objects accurately is non-trivial, especially in challenging scenes. To this end, we develop underlineSAM with seunderlinemantic funderlineeature fuunderlinesion guidancunderlinee (Sammese), which incorporates multi-modal saliency-specific knowledge into SAM to adapt SAM to multi-modal SOD tasks. However, it is difficult for SAM trained on single-modal data to directly mine the complementary benefits of multi-modal inputs and comprehensively utilize them to achieve accurate saliency prediction. To address these issues, we first design a multi-modal complementary fusion module to extract robust multi-modal semantic features by integrating information from visible and thermal or depth image pairs. Then, we feed the extracted multi-modal semantic features into both the SAM image encoder and mask decoder for fine-tuning and prompting, respectively. Specifically, in the image encoder, a multi-modal adapter is proposed to adapt the single-modal SAM to multi-modal information. In the mask decoder, a semantic-geometric prompt generation strategy is proposed to produce corresponding embeddings with various saliency cues. Extensive experiments on both RGB-D and RGB-T SOD benchmarks show the effectiveness of the proposed framework. The code will be available at urlhttps://github.com/Angknpng/Sammese.