# Transformer-Based Source-Free Domain Adaptation

arXiv: https://arxiv.org/abs/2105.14138 · HTML: https://mlsanity.com/paper/2105.14138/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 73 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [Unsupervised Domain Adaptation with Residual Transfer Networks](https://mlsanity.com/paper/1602.04433/) (2016)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Learning to Learn Single Domain Generalization](https://mlsanity.com/paper/2003.13216/) (2020)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)

## Built on by (most cited first, 30 of 30)

- [CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2109.06165/) (2021)
- [A Comprehensive Survey on Source-free Domain Adaptation](https://mlsanity.com/paper/2302.11803/) (2023)
- [Source-Free Unsupervised Domain Adaptation: A Survey](https://mlsanity.com/paper/2301.00265/) (2022)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalization](https://mlsanity.com/paper/2306.01103/) (2023)
- [Looking Outside the Window: Wide-Context Transformer for the Semantic Segmentation of High-Resolution Remote Sensing Images](https://mlsanity.com/paper/2106.15754/) (2021)
- [CoNMix for Source-free Single and Multi-target Domain Adaptation](https://mlsanity.com/paper/2211.03876/) (2022)
- [Active Test-Time Adaptation: Theoretical Analyses and An Algorithm](https://mlsanity.com/paper/2404.05094/) (2024)
- [Domain-Specificity Inducing Transformers for Source-Free Domain Adaptation](https://mlsanity.com/paper/2308.14023/) (2023)
- [Domain Adaptation via Bidirectional Cross-Attention Transformer](https://mlsanity.com/paper/2201.05887/) (2022)
- [Vision transformers in domain adaptation and domain generalization: a study of robustness](https://mlsanity.com/paper/2404.04452/) (2024)
- [Unsupervised Domain Adaptation for Video Transformers in Action Recognition](https://mlsanity.com/paper/2207.12842/) (2022)
- [Towards Source-free Domain Adaptive Semantic Segmentation via Importance-aware and Prototype-contrast Learning](https://mlsanity.com/paper/2306.01598/) (2023)
- [Source-Free Domain Adaptation of Weakly-Supervised Object Localization Models for Histology](https://mlsanity.com/paper/2404.19113/) (2024)
- [Disentangled Source-Free Personalization for Facial Expression Recognition with Neutral Target Data](https://mlsanity.com/paper/2503.20771/) (2025)
- [Exploiting Both Domain-specific and Invariant Knowledge via a Win-win Transformer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2111.12941/) (2021)
- [Towards Unsupervised Domain Adaptation via Domain-Transformer](https://mlsanity.com/paper/2202.13777/) (2022)
- [Making the Best of Both Worlds: A Domain-Oriented Transformer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2208.01195/) (2022)
- [Boosting Few-shot Semantic Segmentation with Transformers](https://mlsanity.com/paper/2108.02266/) (2021)
- [Aligning Non-Causal Factors for Transformer-Based Source-Free Domain Adaptation](https://mlsanity.com/paper/2311.16294/) (2023)
- [Label Calibration in Source Free Domain Adaptation](https://mlsanity.com/paper/2501.07072/) (2025)
- [Unsupervised Domain Adaptation via Style-Aware Self-intermediate Domain](https://mlsanity.com/paper/2209.01870/) (2022)
- [Key Design Choices in Source-Free Unsupervised Domain Adaptation: An In-depth Empirical Analysis](https://mlsanity.com/paper/2402.16090/) (2024)
- [HyDA: Hypernetworks for Test Time Domain Adaptation in Medical Imaging Analysis](https://mlsanity.com/paper/2503.04979/) (2025)
- [Reiterative Domain Aware Multi-Target Adaptation](https://mlsanity.com/paper/2109.00919/) (2021)
- [Model-Contrastive Federated Domain Adaptation](https://mlsanity.com/paper/2305.10432/) (2023)
- [Trust And Balance: Few Trusted Samples Pseudo-Labeling and Temperature Scaled Loss for Effective Source-Free Unsupervised Domain Adaptation](https://mlsanity.com/paper/2409.00741/) (2024)
- [LEARN: A Unified Framework for Multi-Task Domain Adapt Few-Shot Learning](https://mlsanity.com/paper/2412.16275/) (2024)
- [Source-Free Domain Adaptation via Multi-view Contrastive Learning](https://mlsanity.com/paper/2507.03321/) (2025)
- [SfMamba: Efficient Source-Free Domain Adaptation via Selective Scan Modeling](https://mlsanity.com/paper/2601.08608/) (2026)

## Abstract (arXiv)

In this paper, we study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. Previous works on SFDA mainly focus on aligning the cross-domain distributions. However, they ignore the generalization ability of the pretrained source model, which largely influences the initial target outputs that are vital to the target adaptation stage. To address this, we make the interesting observation that the model accuracy is highly correlated with whether or not attention is focused on the objects in an image. To this end, we propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA. Specifically, we apply the Transformer as the attention module and inject it into a convolutional network. By doing so, the model is encouraged to turn attention towards the object regions, which can effectively improve the model's generalization ability on the target domains. Moreover, a novel self-supervised knowledge distillation approach is proposed to adapt the Transformer with target pseudo-labels, thus further encouraging the network to focus on the object regions. Experiments on three domain adaptation tasks, including closed-set, partial-set, and open-set adaption, demonstrate that TransDA can greatly improve the adaptation accuracy and produce state-of-the-art results. The source code and trained models are available at https://github.com/ygjwd12345/TransDA.