# TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation

arXiv: https://arxiv.org/abs/2108.05988 · HTML: https://mlsanity.com/paper/2108.05988/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 68 references that are in the map.
- First posted on arXiv 12 Aug 2021.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [Semi-supervised Domain Adaptation via Minimax Entropy](https://mlsanity.com/paper/1904.06487/) (2019)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)

## Built on by (most cited first, 30 of 81)

- [MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation](https://mlsanity.com/paper/2212.01322/) (2022)
- [CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2109.06165/) (2021)
- [A Closer Look at Smoothness in Domain Adversarial Training](https://mlsanity.com/paper/2206.08213/) (2022)
- [AD-CLIP: Adapting Domains in Prompt Space Using CLIP](https://mlsanity.com/paper/2308.05659/) (2023)
- [Safe Self-Refinement for Transformer-based Domain Adaptation](https://mlsanity.com/paper/2204.07683/) (2022)
- [Patch-Mix Transformer for Unsupervised Domain Adaptation: A Game Perspective](https://mlsanity.com/paper/2303.13434/) (2023)
- [MTTrans: Cross-Domain Object Detection with Mean-Teacher Transformer](https://mlsanity.com/paper/2205.01643/) (2022)
- [Free Lunch for Domain Adversarial Training: Environment Label Smoothing](https://mlsanity.com/paper/2302.00194/) (2023)
- [Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2403.02899/) (2024)
- [Periodically Exchange Teacher-Student for Source-Free Object Detection](https://mlsanity.com/paper/2311.13930/) (2023)
- [Domain-Specificity Inducing Transformers for Source-Free Domain Adaptation](https://mlsanity.com/paper/2308.14023/) (2023)
- [ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts](https://mlsanity.com/paper/2406.10973/) (2024)
- [Universal Domain Adaptation via Compressive Attention Matching](https://mlsanity.com/paper/2304.11862/) (2023)
- [Domain Adaptation via Bidirectional Cross-Attention Transformer](https://mlsanity.com/paper/2201.05887/) (2022)
- [Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2403.06946/) (2024)
- [Unsupervised Domain Adaption Harnessing Vision-Language Pre-training](https://mlsanity.com/paper/2408.02192/) (2024)
- [Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency](https://mlsanity.com/paper/2206.08222/) (2022)
- [Black-box Unsupervised Domain Adaptation with Bi-directional Atkinson-Shiffrin Memory](https://mlsanity.com/paper/2308.13236/) (2023)
- [AGI for Agriculture](https://mlsanity.com/paper/2304.06136/) (2023)
- [A Survey on Deep Clustering: From the Prior Perspective](https://mlsanity.com/paper/2406.19602/) (2024)
- [Gradient Harmonization in Unsupervised Domain Adaptation](https://mlsanity.com/paper/2408.00288/) (2024)
- [DomainForensics: Exposing Face Forgery across Domains via Bi-directional Adaptation](https://mlsanity.com/paper/2312.10680/) (2023)
- [Towards Unsupervised Domain Adaptation via Domain-Transformer](https://mlsanity.com/paper/2202.13777/) (2022)
- [Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Data](https://mlsanity.com/paper/2311.01420/) (2023)
- [Cross-Domain Few-Shot Learning via Adaptive Transformer Networks](https://mlsanity.com/paper/2401.13987/) (2024)
- [Generalizable Facial Expression Recognition](https://mlsanity.com/paper/2408.10614/) (2024)
- [Simulations of Common Unsupervised Domain Adaptation Algorithms for Image Classification](https://mlsanity.com/paper/2502.10694/) (2025)
- [NoisyNN: Exploring the Impact of Information Entropy Change in Learning Systems](https://mlsanity.com/paper/2309.10625/) (2023)
- [Aligning Non-Causal Factors for Transformer-Based Source-Free Domain Adaptation](https://mlsanity.com/paper/2311.16294/) (2023)
- [CLIP the Divergence: Language-guided Unsupervised Domain Adaptation](https://mlsanity.com/paper/2407.01842/) (2024)

## Abstract (arXiv)

Unsupervised domain adaptation (UDA) aims to transfer the knowledge learnt from a labeled source domain to an unlabeled target domain. Previous work is mainly built upon convolutional neural networks (CNNs) to learn domain-invariant representations. With the recent exponential increase in applying Vision Transformer (ViT) to vision tasks, the capability of ViT in adapting cross-domain knowledge, however, remains unexplored in the literature. To fill this gap, this paper first comprehensively investigates the transferability of ViT on a variety of domain adaptation tasks. Surprisingly, ViT demonstrates superior transferability over its CNNs-based counterparts with a large margin, while the performance can be further improved by incorporating adversarial adaptation. Notwithstanding, directly using CNNs-based adaptation strategies fails to take the advantage of ViT's intrinsic merits (e.g., attention mechanism and sequential image representation) which play an important role in knowledge transfer. To remedy this, we propose an unified framework, namely Transferable Vision Transformer (TVT), to fully exploit the transferability of ViT for domain adaptation. Specifically, we delicately devise a novel and effective unit, which we term Transferability Adaption Module (TAM). By injecting learned transferabilities into attention blocks, TAM compels ViT focus on both transferable and discriminative features. Besides, we leverage discriminative clustering to enhance feature diversity and separation which are undermined during adversarial domain alignment. To verify its versatility, we perform extensive studies of TVT on four benchmarks and the experimental results demonstrate that TVT attains significant improvements compared to existing state-of-the-art UDA methods.