# Few-Shot Domain Adaptation with Polymorphic Transformers

arXiv: https://arxiv.org/abs/2107.04805 · HTML: https://mlsanity.com/paper/2107.04805/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 10 Jul 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions](https://mlsanity.com/paper/1812.03664/) (2018)
- [REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs](https://mlsanity.com/paper/1910.03667/) (2019)
- [Few-Shot Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.02536/) (2017)
- [Unsupervised Cross-Modality Domain Adaptation of ConvNets for Biomedical Image Segmentations with Adversarial Loss](https://mlsanity.com/paper/1804.10916/) (2018)
- [Medical Image Segmentation Using Squeeze-and-Expansion Transformers](https://mlsanity.com/paper/2105.09511/) (2021)
- [Patch-based Output Space Adversarial Learning for Joint Optic Disc and Cup Segmentation](https://mlsanity.com/paper/1902.07519/) (2019)
- [Dual-Teacher: Integrating Intra-domain and Inter-domain Teachers for Annotation-efficient Cardiac Segmentation](https://mlsanity.com/paper/2007.06279/) (2020)
- [Unsupervised Domain Adaptation for Automatic Estimation of Cardiothoracic Ratio](https://mlsanity.com/paper/1807.03434/) (2018)
- [A Strong Baseline for Domain Adaptation and Generalization in Medical Imaging](https://mlsanity.com/paper/1904.01638/) (2019)

## Built on by (most cited first, 14 of 14)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [REFUGE2 Challenge: A Treasure Trove for Multi-Dimension Analysis and Evaluation in Glaucoma Screening](https://mlsanity.com/paper/2202.08994/) (2022)
- [A Systematic Review of Few-Shot Learning in Medical Imaging](https://mlsanity.com/paper/2309.11433/) (2023)
- [Semantic Image Segmentation: Two Decades of Research](https://mlsanity.com/paper/2302.06378/) (2023)
- [Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning](https://mlsanity.com/paper/2205.12466/) (2022)
- [Optimizing Vision Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2210.08066/) (2022)
- [RobustEMD: Domain Robust Matching for Cross-domain Few-shot Medical Image Segmentation](https://mlsanity.com/paper/2410.01110/) (2024)
- [Colorectal Polyp Segmentation in the Deep Learning Era: A Comprehensive Survey](https://mlsanity.com/paper/2401.11734/) (2024)
- [A Survey of Deep Visual Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2303.09253/) (2023)
- [INDIGO: Intrinsic Multimodality for Domain Generalization](https://mlsanity.com/paper/2206.05912/) (2022)
- [Unsupervised Domain Adaptive Fundus Image Segmentation with Few Labeled Source Data](https://mlsanity.com/paper/2210.04379/) (2022)
- [CASHformer: Cognition Aware SHape Transformer for Longitudinal Analysis](https://mlsanity.com/paper/2207.02091/) (2022)
- [Few-Shot Domain Adaptation for Learned Image Compression](https://mlsanity.com/paper/2409.11111/) (2024)
- [Meta-learners for few-shot weakly-supervised optic disc and cup segmentation on fundus images](https://mlsanity.com/paper/2512.15061/) (2025)

## Abstract (arXiv)

Deep neural networks (DNNs) trained on one set of medical images often experience severe performance drop on unseen test images, due to various domain discrepancy between the training images (source domain) and the test images (target domain), which raises a domain adaptation issue. In clinical settings, it is difficult to collect enough annotated target domain data in a short period. Few-shot domain adaptation, i.e., adapting a trained model with a handful of annotations, is highly practical and useful in this case. In this paper, we propose a Polymorphic Transformer (Polyformer), which can be incorporated into any DNN backbones for few-shot domain adaptation. Specifically, after the polyformer layer is inserted into a model trained on the source domain, it extracts a set of prototype embeddings, which can be viewed as a "basis" of the source-domain features. On the target domain, the polyformer layer adapts by only updating a projection layer which controls the interactions between image features and the prototype embeddings. All other model weights (except BatchNorm parameters) are frozen during adaptation. Thus, the chance of overfitting the annotations is greatly reduced, and the model can perform robustly on the target domain after being trained on a few annotated images. We demonstrate the effectiveness of Polyformer on two medical segmentation tasks (i.e., optic disc/cup segmentation, and polyp segmentation). The source code of Polyformer is released at https://github.com/askerlee/segtran.