Pre-Training Transformers for Domain Adaptation
Not on the map. Why?
Abstract (arXiv)
The Visual Domain Adaptation Challenge 2021 called for unsupervised domain adaptation methods that could improve the performance of models by transferring the knowledge obtained from source datasets to out-of-distribution target datasets. In this paper, we utilize BeiT [1] and demonstrate its capability of capturing key attributes from source datasets and apply it to target datasets in a semi-supervised manner. Our method was able to outperform current state-of-the-art (SoTA) techniques and was able to achieve 1st place on the ViSDA Domain Adaptation Challenge with ACC of 56.29% and AUROC of 69.79%.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 19 of its 23 references that are in the map. First posted on arXiv 18 Dec 2021.
Builds on
19 of 23 references.
References · 19
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
- 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
- 2017Adversarial Discriminative Domain Adaptationcited by 1,309
- 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
7 more
- 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
- 2017Dynamic Routing Between Capsulescited by 689
- 2019Contrastive Adaptation Network for Unsupervised Domain Adaptationcited by 407
- 2020Unsupervised Domain Adaptation via Structurally Regularized Deep Clusteringcited by 140
- 2021VisDA-2021 Competition Universal Domain Adaptation to Improve Performance on Out-of-Distribution Datacited by 19
- 2021Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domainscited by 14
- 2021Exploiting Image Translations via Ensemble Self-Supervised Learning for Unsupervised Domain Adaptationcited by 6
Built on by
Citing papers per year: 2024: 1.
1 paper.