# DFR: A Decompose-Fuse-Reconstruct Framework for Multi-Modal Few-Shot Segmentation

arXiv: https://arxiv.org/abs/2507.16736 · HTML: https://mlsanity.com/paper/2507.16736/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 22 Jul 2025.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [Hypercorrelation Squeeze for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01538/) (2021)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching](https://mlsanity.com/paper/2305.13310/) (2023)
- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer](https://mlsanity.com/paper/2309.07929/) (2023)
- [Adapt Before Comparison: A New Perspective on Cross-Domain Few-Shot Segmentation](https://mlsanity.com/paper/2402.17614/) (2024)
- [SSformer: A Lightweight Transformer for Semantic Segmentation](https://mlsanity.com/paper/2208.02034/) (2022)
- [Relevant Intrinsic Feature Enhancement Network for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2312.06474/) (2023)
- [Adaptive FSS: A Novel Few-Shot Segmentation Framework via Prototype Enhancement](https://mlsanity.com/paper/2312.15731/) (2023)
- [Foreground-Covering Prototype Generation and Matching for SAM-Aided Few-Shot Segmentation](https://mlsanity.com/paper/2501.00752/) (2025)
- [AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation](https://mlsanity.com/paper/2408.01708/) (2024)
- [Extending Segment Anything Model into Auditory and Temporal Dimensions for Audio-Visual Segmentation](https://mlsanity.com/paper/2406.06163/) (2024)

## Abstract (arXiv)

This paper presents DFR (Decompose, Fuse and Reconstruct), a novel framework that addresses the fundamental challenge of effectively utilizing multi-modal guidance in few-shot segmentation (FSS). While existing approaches primarily rely on visual support samples or textual descriptions, their single or dual-modal paradigms limit exploitation of rich perceptual information available in real-world scenarios. To overcome this limitation, the proposed approach leverages the Segment Anything Model (SAM) to systematically integrate visual, textual, and audio modalities for enhanced semantic understanding. The DFR framework introduces three key innovations: 1) Multi-modal Decompose: a hierarchical decomposition scheme that extracts visual region proposals via SAM, expands textual semantics into fine-grained descriptors, and processes audio features for contextual enrichment; 2) Multi-modal Contrastive Fuse: a fusion strategy employing contrastive learning to maintain consistency across visual, textual, and audio modalities while enabling dynamic semantic interactions between foreground and background features; 3) Dual-path Reconstruct: an adaptive integration mechanism combining semantic guidance from tri-modal fused tokens with geometric cues from multi-modal location priors. Extensive experiments across visual, textual, and audio modalities under both synthetic and real settings demonstrate DFR's substantial performance improvements over state-of-the-art methods.