# Object Segmentation by Mining Cross-Modal Semantics

arXiv: https://arxiv.org/abs/2305.10469 · HTML: https://mlsanity.com/paper/2305.10469/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 92 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Anabranch Network for Camouflaged Object Segmentation](https://mlsanity.com/paper/2105.09451/) (2021)
- [Simultaneously Localize, Segment and Rank the Camouflaged Objects](https://mlsanity.com/paper/2103.04011/) (2021)
- [Zoom In and Out: A Mixed-scale Triplet Network for Camouflaged Object Detection](https://mlsanity.com/paper/2203.02688/) (2022)
- [FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation](https://mlsanity.com/paper/2103.02242/) (2021)
- [Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks](https://mlsanity.com/paper/1907.06781/) (2019)
- [Uncertainty-aware Joint Salient Object and Camouflaged Object Detection](https://mlsanity.com/paper/2104.02628/) (2021)
- [Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging](https://mlsanity.com/paper/2105.14021/) (2021)
- [JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection](https://mlsanity.com/paper/2004.08515/) (2020)
- [Specificity-preserving RGB-D Saliency Detection](https://mlsanity.com/paper/2108.08162/) (2021)
- [RGBT Salient Object Detection: A Large-scale Dataset and Benchmark](https://mlsanity.com/paper/2007.03262/) (2020)
- [From Denoising to Compressed Sensing](https://mlsanity.com/paper/1406.4175/) (2014)
- [SwinNet: Swin Transformer drives edge-aware RGB-D and RGB-T salient object detection](https://mlsanity.com/paper/2204.05585/) (2022)
- [RGB-T Image Saliency Detection via Collaborative Graph Learning](https://mlsanity.com/paper/1905.06741/) (2019)
- [CamoFormer: Masked Separable Attention for Camouflaged Object Detection](https://mlsanity.com/paper/2212.06570/) (2022)
- [Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion](https://mlsanity.com/paper/2103.11832/) (2021)
- [Source-free Depth for Object Pop-out](https://mlsanity.com/paper/2212.05370/) (2022)
- [A Unified RGB-T Saliency Detection Benchmark: Dataset, Baselines, Analysis and A Novel Approach](https://mlsanity.com/paper/1701.02829/) (2017)
- [RGB-D Saliency Detection via Cascaded Mutual Information Minimization](https://mlsanity.com/paper/2109.07246/) (2021)
- [Cross-Modal Weighting Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2007.04901/) (2020)
- [TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network](https://mlsanity.com/paper/2108.03990/) (2021)
- [Multi-interactive Dual-decoder for RGB-thermal Salient Object Detection](https://mlsanity.com/paper/2005.02315/) (2020)

## Built on by (most cited first, 12 of 12)

- [A Survey of Camouflaged Object Detection and Beyond](https://mlsanity.com/paper/2408.14562/) (2024)
- [Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2410.22489/) (2024)
- [FocusDiffuser: Perceiving Local Disparities for Camouflaged Object Detection](https://mlsanity.com/paper/2407.13133/) (2024)
- [Lightweight RGB-D Salient Object Detection from a Speed-Accuracy Tradeoff Perspective](https://mlsanity.com/paper/2505.04758/) (2025)
- [Cross-View Multi-Modal Segmentation @ Ego-Exo4D Challenges 2025](https://mlsanity.com/paper/2506.05856/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [Stepwise Decomposition and Dual-stream Focus: A Novel Approach for Training-free Camouflaged Object Segmentation](https://mlsanity.com/paper/2506.06818/) (2025)
- [SAMSOD: Rethinking SAM Optimization for RGB-T Salient Object Detection](https://mlsanity.com/paper/2510.03689/) (2025)
- [LEAF-Mamba: Local Emphatic and Adaptive Fusion State Space Model for RGB-D Salient Object Detection](https://mlsanity.com/paper/2509.18683/) (2025)
- [RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation](https://mlsanity.com/paper/2609.12622/) (2026)
- [IV-tuning: Parameter-Efficient Transfer Learning for Infrared-Visible Tasks](https://mlsanity.com/paper/2412.16654/) (2024)
- [HiddenObject: Modality-Agnostic Fusion for Multimodal Hidden Object Detection](https://mlsanity.com/paper/2508.21135/) (2025)

## Abstract (arXiv)

Multi-sensor clues have shown promise for object segmentation, but inherent noise in each sensor, as well as the calibration error in practice, may bias the segmentation accuracy. In this paper, we propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features, with the aim of controlling the modal contribution based on relative entropy. We explore semantics among the multimodal inputs in two aspects: the modality-shared consistency and the modality-specific variation. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision. On the one hand, the AF block explicitly dissociates the shared and specific representation and learns to weight the modal contribution by adjusting the textitproportion, region, and textitpattern, depending upon the quality. On the other hand, our CFD initially decodes the shared feature and then refines the output through specificity-aware querying. Further, we enforce semantic consistency across the decoding layers to enable interaction across network hierarchies, improving feature discriminability. Exhaustive comparison on eleven datasets with depth or thermal clues, and on two challenging tasks, namely salient and camouflage object segmentation, validate our effectiveness in terms of both performance and robustness. The source code is publicly available at https://github.com/Zongwei97/XMSNet.