# Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation

arXiv: https://arxiv.org/abs/2504.03193 · HTML: https://mlsanity.com/paper/2504.03193/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 61 of its 72 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)
- State-space sequence models (S4, Mamba) / [Vision Mamba backbones](https://mlsanity.com/ssm/vision-mamba-backbones/)

## Builds on (most cited first, 30 of 61)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers](https://mlsanity.com/paper/2208.06366/) (2022)
- [VM-UNet: Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2402.02491/) (2024)
- [Long Range Language Modeling via Gated State Spaces](https://mlsanity.com/paper/2206.13947/) (2022)
- [MambaVision: A Hybrid Mamba-Transformer Vision Backbone](https://mlsanity.com/paper/2407.08083/) (2024)
- [RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening](https://mlsanity.com/paper/2103.15597/) (2021)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.13132/) (2022)

## Built on by (most cited first, 24 of 24)

- [HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation](https://mlsanity.com/paper/2507.15542/) (2025)
- [RGB-to-Polarization Estimation: A New Task and Benchmark Study](https://mlsanity.com/paper/2505.13050/) (2025)
- [Bridging Day and Night: Target-Class Hallucination Suppression in Unpaired Image Translation](https://mlsanity.com/paper/2602.15383/) (2026)
- [Exploring Probabilistic Modeling Beyond Domain Generalization for Semantic Segmentation](https://mlsanity.com/paper/2507.21367/) (2025)
- [ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation](https://mlsanity.com/paper/2509.00665/) (2025)
- [Rein++: Efficient Generalization and Adaptation for Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2508.01667/) (2025)
- [Lightweight Quad Bayer HybridEVS Demosaicing via State Space Augmented Cross-Attention](https://mlsanity.com/paper/2508.06058/) (2025)
- [Causal-Tune: Mining Causal Factors from Vision Foundation Models for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2512.16567/) (2025)
- [GlowGS: Generative Semantic Feature Learning for 3D Gaussian Splatting in Nighttime Glow Scenes](https://mlsanity.com/paper/2605.23602/) (2026)
- [UAM: A Unified Attention-Mamba Backbone of Multimodal Framework for Tumor Cell Classification](https://mlsanity.com/paper/2511.17355/) (2025)
- [Partial Ring Scan: Revisiting Scan Order in Vision State Space Models](https://mlsanity.com/paper/2602.04170/) (2026)
- [Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPO](https://mlsanity.com/paper/2607.11581/) (2026)
- [IELDG: Suppressing Domain-Specific Noise with Inverse Evolution Layers for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2508.19604/) (2025)
- [Vision-Language Semantic Aggregation Leveraging Foundation Model for Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2509.08570/) (2025)
- [Domain Generalization for Semantic Segmentation: A Survey](https://mlsanity.com/paper/2510.03540/) (2025)
- [OT-Drive: Out-of-Distribution Off-Road Traversable Area Segmentation via Optimal Transport](https://mlsanity.com/paper/2601.09952/) (2026)
- [DCCS-Det: Directional Context and Cross-Scale-Aware Detector for Infrared Small Target](https://mlsanity.com/paper/2601.16428/) (2026)
- [RecycleLoRA: Rank-Revealing QR-Based Dual-LoRA Subspace Adaptation for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2603.28142/) (2026)
- [OV-Stitcher: A Global Context-Aware Framework for Training-Free Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2604.08110/) (2026)
- [CNS-Edit++: Category-Agnostic 3D Editing with Coupled Neural Shape Representation](https://mlsanity.com/paper/2607.16577/) (2026)
- [UC-VLM: Consistency-Driven Learning for AI-Generated Image Detection with Vision-Language Large Models](https://mlsanity.com/paper/2608.15238/) (2026)
- [Hierarchical Prompt Injector for Domain Generalization Segmentation](https://mlsanity.com/paper/2609.05864/) (2026)
- [Predicting Multi-View Rashomon Representation: Can We Learn Where Models Disagree?](https://mlsanity.com/paper/2609.39848/) (2026)
- [Sparse All-Layer Connector for Domain Generalised Semantic Segmentation](https://mlsanity.com/paper/2610.15660/) (2026)

## Abstract (arXiv)

Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) have gained traction in Domain Generalized Semantic Segmentation (DGSS) due to their strong generalization capabilities. However, existing DGSS methods often rely exclusively on either VFMs or VLMs, overlooking their complementary strengths. VFMs (e.g., DINOv2) excel at capturing fine-grained features, while VLMs (e.g., CLIP) provide robust text alignment but struggle with coarse granularity. Despite their complementary strengths, effectively integrating VFMs and VLMs with attention mechanisms is challenging, as the increased patch tokens complicate long-sequence modeling. To address this, we propose MFuser, a novel Mamba-based fusion framework that efficiently combines the strengths of VFMs and VLMs while maintaining linear scalability in sequence length. MFuser consists of two key components: MVFuser, which acts as a co-adapter to jointly fine-tune the two models by capturing both sequential and spatial dynamics; and MTEnhancer, a hybrid attention-Mamba module that refines text embeddings by incorporating image priors. Our approach achieves precise feature locality and strong text alignment without incurring significant computational overhead. Extensive experiments demonstrate that MFuser significantly outperforms state-of-the-art DGSS methods, achieving 68.20 mIoU on synthetic-to-real and 71.87 mIoU on real-to-real benchmarks. The code is available at https://github.com/devinxzhang/MFuser.