# VLMs meet UDA: Boosting Transferability of Open Vocabulary Segmentation with Unsupervised Domain Adaptation

arXiv: https://arxiv.org/abs/2412.09240 · HTML: https://mlsanity.com/paper/2412.09240/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 71 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 6 of 6)

- [A Closer Look at the Explainability of Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2304.05653/) (2023)
- [CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks](https://mlsanity.com/paper/2206.09059/) (2022)
- [Don't Stop Learning: Towards Continual Learning for the CLIP Model](https://mlsanity.com/paper/2207.09248/) (2022)
- [Collaborating Foundation Models for Domain Generalized Semantic Segmentation](https://mlsanity.com/paper/2312.09788/) (2023)
- [Exploring Visual Interpretability for Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2209.07046/) (2022)
- [Generative Negative Text Replay for Continual Vision-Language Pretraining](https://mlsanity.com/paper/2210.17322/) (2022)

## Built on by (most cited first, 1 of 1)

- [GBT-SAM: A Parameter-Efficient Depth-Aware Model for Generalizable Brain tumour Segmentation on mp-MRI](https://mlsanity.com/paper/2503.04325/) (2025)

## Abstract (arXiv)

Segmentation models are typically constrained by the categories defined during training. To address this, researchers have explored two independent approaches: adapting Vision-Language Models (VLMs) and leveraging synthetic data. However, VLMs often struggle with granularity, failing to disentangle fine-grained concepts, while synthetic data-based methods remain limited by the scope of available datasets. This paper proposes enhancing segmentation accuracy across diverse domains by integrating Vision-Language reasoning with key strategies for Unsupervised Domain Adaptation (UDA). First, we improve the fine-grained segmentation capabilities of VLMs through multi-scale contextual data, robust text embeddings with prompt augmentation, and layer-wise fine-tuning in our proposed Foundational-Retaining Open Vocabulary Semantic Segmentation (FROVSS) framework. Next, we incorporate these enhancements into a UDA framework by employing distillation to stabilize training and cross-domain mixed sampling to boost adaptability without compromising generalization. The resulting UDA-FROVSS framework is the first UDA approach to effectively adapt across domains without requiring shared categories.