# DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance

arXiv: https://arxiv.org/abs/2606.08906 · HTML: https://mlsanity.com/paper/2606.08906/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 124 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Builds on (most cited first, 30 of 61)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Structure-measure: A New Way to Evaluate Foreground Maps](https://mlsanity.com/paper/1708.00786/) (2017)
- [Enhanced-alignment Measure for Binary Foreground Map Evaluation](https://mlsanity.com/paper/1805.10421/) (2018)
- [Anabranch Network for Camouflaged Object Segmentation](https://mlsanity.com/paper/2105.09451/) (2021)
- [A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images](https://mlsanity.com/paper/1612.00799/) (2016)
- [Simultaneously Localize, Segment and Rank the Camouflaged Objects](https://mlsanity.com/paper/2103.04011/) (2021)
- [Visual Saliency Based on Multiscale Deep Features](https://mlsanity.com/paper/1503.08663/) (2015)
- [Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Images](https://mlsanity.com/paper/2004.14133/) (2020)
- [Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.09910/) (2022)
- [Visual Saliency Transformer](https://mlsanity.com/paper/2104.12099/) (2021)
- [Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal](https://mlsanity.com/paper/1712.02478/) (2017)
- [Explicit Visual Prompting for Low-Level Structure Segmentations](https://mlsanity.com/paper/2303.10883/) (2023)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos](https://mlsanity.com/paper/1604.00136/) (2016)
- [RGBT Salient Object Detection: A Large-scale Dataset and Benchmark](https://mlsanity.com/paper/2007.03262/) (2020)
- [SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation](https://mlsanity.com/paper/2408.08870/) (2024)
- [Implicit Motion Handling for Video Camouflaged Object Detection](https://mlsanity.com/paper/2203.07363/) (2022)
- [SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More](https://mlsanity.com/paper/2408.04579/) (2024)
- [RGB-T Image Saliency Detection via Collaborative Graph Learning](https://mlsanity.com/paper/1905.06741/) (2019)

## Built on by (most cited first, 6 of 6)

- [Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation](https://mlsanity.com/paper/2607.21973/) (2026)
- [HVPNet: A Bio-Inspired Network for General Salient and Camouflaged Object Detection](https://mlsanity.com/paper/2606.31496/) (2026)
- [TDFNet: Tri-projection Deformable Fusion Network for Panoramic Salient Object Detection](https://mlsanity.com/paper/2608.25808/) (2026)
- [ViCo-SAM3: Vision-Conditioned Alignment for Open-Vocabulary Camouflaged Object Segmentation](https://mlsanity.com/paper/2609.15418/) (2026)
- [Understanding and Overcoming Cross-modal Fusion Bias in Multimodal Anomaly Detection From A Fisher Information Perspective](https://mlsanity.com/paper/2608.00986/) (2026)
- [S2A:Semantic-to-Spatial Alignment for Alignment-Free RGB-T Salient Object Detection](https://mlsanity.com/paper/2609.27413/) (2026)

## Abstract (arXiv)

In many binary segmentation tasks, most multimodal methods rely on fixed feature concatenation for cross-modal interaction and straightforward decoder designs dominated by low-frequency semantics. However, they ignore two key challenges: one is the lack of an adaptive mechanism to handle modality discrepancies and complementarity, and the other is the absence of an efficient decoding strategy to balance both high- and low-frequency representations. In this work, we propose a simple yet general multimodal binary segmentation framework, termed DifferSeg, to address both problems simultaneously. With the help of the differential perception fusion (DPF) module, DifferSeg employs learnable differential operators to adaptively align multimodal features and enhance their complementarity through residual fusion, effectively mitigating modality mismatch and fusion redundancy. In addition, we design a frequency-guided decoder (FGD) that builds cross-frequency interactions and multi-path upsampling to maintain consistency between detailed high-frequency structures and semantic low-frequency representations, ensuring fine-grained boundary recovery and noise suppression. % Benefiting from these designs, DifferSeg can be easily generalized to diverse binary segmentation tasks, including both natural and medical modalities. Without bells and whistles, it consistently surpasses 67 state-of-the-art methods across 29 public datasets involving 18 downstream tasks, demonstrating superior generalization and segmentation accuracy.Code and pretrained models will be available at the Link.