# Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation

arXiv: https://arxiv.org/abs/2312.03502 · HTML: https://mlsanity.com/paper/2312.03502/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 78 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 30 of 54)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [End-to-End Semi-Supervised Object Detection with Soft Teacher](https://mlsanity.com/paper/2106.09018/) (2021)
- [A Simple Semi-Supervised Learning Framework for Object Detection](https://mlsanity.com/paper/2005.04757/) (2020)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [Anabranch Network for Camouflaged Object Segmentation](https://mlsanity.com/paper/2105.09451/) (2021)
- [Universal Source-Free Domain Adaptation](https://mlsanity.com/paper/2004.04393/) (2020)
- [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016)
- [Generalized Source-free Domain Adaptation](https://mlsanity.com/paper/2108.01614/) (2021)
- [Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer](https://mlsanity.com/paper/2012.07297/) (2020)
- [Source-Free Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16372/) (2021)
- [Category Anchor-Guided Unsupervised Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/1910.13049/) (2019)

## Built on by (most cited first, 30 of 31)

- [Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models](https://mlsanity.com/paper/2403.17589/) (2024)
- [PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Images](https://mlsanity.com/paper/2409.13401/) (2024)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation](https://mlsanity.com/paper/2503.17940/) (2025)
- [Segment Anything with Multiple Modalities](https://mlsanity.com/paper/2408.09085/) (2024)
- [Promptable Anomaly Segmentation with SAM Through Self-Perception Tuning](https://mlsanity.com/paper/2411.17217/) (2024)
- [Mamba-Sea: A Mamba-based Framework with Global-to-Local Sequence Augmentation for Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2504.17515/) (2025)
- [OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triad](https://mlsanity.com/paper/2503.18695/) (2025)
- [Bootstrap Segmentation Foundation Model under Distribution Shift via Object-Centric Learning](https://mlsanity.com/paper/2408.16310/) (2024)
- [Enhancing Skin Disease Diagnosis: Interpretable Visual Concept Discovery with SAM](https://mlsanity.com/paper/2409.09520/) (2024)
- [Exploring Probabilistic Modeling Beyond Domain Generalization for Semantic Segmentation](https://mlsanity.com/paper/2507.21367/) (2025)
- [SAMIC: Segment Anything with In-Context Spatial Prompt Engineering](https://mlsanity.com/paper/2412.11998/) (2024)
- [AIF-SFDA: Autonomous Information Filter-driven Source-Free Domain Adaptation for Medical Image Segmentation](https://mlsanity.com/paper/2501.03074/) (2025)
- [SAM-DA: Decoder Adapter for Efficient Medical Domain Adaptation](https://mlsanity.com/paper/2501.06836/) (2025)
- [SAM-aware Test-time Adaptation for Universal Medical Image Segmentation](https://mlsanity.com/paper/2506.05221/) (2025)
- [Beyond Boundaries: Leveraging Vision Foundation Models for Source-Free Object Detection](https://mlsanity.com/paper/2511.07301/) (2025)
- [Adapting Segment Anything Model for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2409.15481/) (2024)
- [Generalizing Segmentation Foundation Model Under Sim-to-real Domain-shift for Guidewire Segmentation in X-ray Fluoroscopy](https://mlsanity.com/paper/2410.07460/) (2024)
- [Using Foundation Models as Pseudo-Label Generators for Pre-Clinical 4D Cardiac CT Segmentation](https://mlsanity.com/paper/2505.09564/) (2025)
- [Prompt-DAS: Annotation-Efficient Prompt Learning for Domain Adaptive Semantic Segmentation of Electron Microscopy Images](https://mlsanity.com/paper/2509.18973/) (2025)
- [ReSAM: Refine, Requery, and Reinforce: Self-Prompting Point-Supervised Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2511.21606/) (2025)
- [RobustMedSAM: Degradation-Resilient Medical Image Segmentation via Robust Foundation Model Adaptation](https://mlsanity.com/paper/2604.09814/) (2026)
- [Intelligent Communication Mixture-of-Experts Boosted-Medical Image Segmentation Foundation Model](https://mlsanity.com/paper/2510.17684/) (2025)
- [BioDet: Boosting Industrial Object Detection with Image Preprocessing Strategies](https://mlsanity.com/paper/2510.21000/) (2025)
- [AD-SAM: Adapting the Segment Anything Model for Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2510.27047/) (2025)
- [Boundary-Aware Test-Time Adaptation for Zero-Shot Medical Image Segmentation](https://mlsanity.com/paper/2512.04520/) (2025)
- [Prefer-DAS: Learning from Local Preferences and Sparse Prompts for Domain Adaptive Segmentation of Electron Microscopy](https://mlsanity.com/paper/2602.19423/) (2026)
- [An Automated Radiomics Framework for Postoperative Survival Prediction in Colorectal Liver Metastases using Preoperative MRI](https://mlsanity.com/paper/2603.10216/) (2026)
- [Do Instance Priors Help Weakly Supervised Semantic Segmentation?](https://mlsanity.com/paper/2604.11170/) (2026)
- [Rethinking Dense Optical Flow without Test-Time Scaling](https://mlsanity.com/paper/2605.08000/) (2026)

## Abstract (arXiv)

The success of large language models has inspired the computer vision community to explore image segmentation foundation model that is able to zero/few-shot generalize through prompt engineering. Segment-Anything(SAM), among others, is the state-of-the-art image segmentation foundation model demonstrating strong zero/few-shot generalization. Despite the success, recent studies reveal the weakness of SAM under strong distribution shift. In particular, SAM performs awkwardly on corrupted natural images, camouflaged images, medical images, etc. Motivated by the observations, we aim to develop a self-training based strategy to adapt SAM to target distribution. Given the unique challenges of large source dataset, high computation cost and incorrect pseudo label, we propose a weakly supervised self-training architecture with anchor regularization and low-rank finetuning to improve the robustness and computation efficiency of adaptation. We validate the effectiveness on 5 types of downstream segmentation tasks including natural clean/corrupted images, medical images, camouflaged images and robotic images. Our proposed method is task-agnostic in nature and outperforms pre-trained SAM and state-of-the-art domain adaptation methods on almost all downstream tasks with the same testing prompt inputs.