mlsanityData to 7 Oct 2026

A Broad Study of Pre-training for Domain Generalization and Adaptation

arXiv 2203.11819 ↗First posted 22 Mar 2022cs.CVcited by 64

Not on the map. Why?

Abstract (arXiv)

Deep models must learn robust and transferable representations in order to perform well on new domains. While domain transfer methods (e.g., domain adaptation, domain generalization) have been proposed to learn transferable representations across domains, they are typically applied to ResNet backbones pre-trained on ImageNet. Thus, existing works pay little attention to the effects of pre-training on domain transfer tasks. In this paper, we provide a broad study and in-depth analysis of pre-training for domain adaptation and generalization, namely: network architectures, size, pre-training loss, and datasets. We observe that simply using a state-of-the-art backbone outperforms existing state-of-the-art domain adaptation baselines and set new baselines on Office-Home and DomainNet improving by 10.7% and 5.5%. We hope that this work can provide more insights for future domain transfer research.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

57 of 74 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2017Squeeze-and-Excitation Networkscited by 4,763
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  13. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  14. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  15. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  16. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  17. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  18. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  19. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  20. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  21. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  22. 2020In Search of Lost Domain Generalizationcited by 1,042
  23. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  24. 2017Maximum Classifier Discrepancy for Unsupervised Domain Adaptationcited by 682
  25. 2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 597
  26. 2018Do Better ImageNet Models Transfer Better?cited by 560
  27. 2016Unsupervised Domain Adaptation with Residual Transfer Networkscited by 340
  28. 2019Semi-supervised Domain Adaptation via Minimax Entropycited by 333
  29. 2019Bridging Theory and Algorithm for Domain Adaptationcited by 332
  30. 2018Strong-Weak Distribution Alignment for Adaptive Object Detectioncited by 269
  31. 2018DeepJDOT: Deep Joint Distribution Optimal Transport for Unsupervised Domain Adaptationcited by 212
  32. 2021Are Transformers More Robust Than CNNs?cited by 187
  33. 2020Universal Domain Adaptation through Self Supervisioncited by 172
  34. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  35. 2019Domain Adaptation for Structured Output via Discriminative Patch Representationscited by 150
  36. 2020Unsupervised Domain Adaptation via Structurally Regularized Deep Clusteringcited by 140
  37. 2018Deep Cocktail Network: Multi-source Unsupervised Domain Adaptation with Category Shiftcited by 109
  38. 2020The iWildCam 2020 Competition Datasetcited by 86

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 25; 2024: 19; 2025: 3; 2026: 5.

64 papers.

Sort:

By area:

  1. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  2. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  3. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learningcited by 58
  4. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  5. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  6. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  7. 2022Towards a Unified View on Visual Parameter-Efficient Transfer Learningcited by 25
  8. 2023ClusT3: Information Invariant Test-Time Trainingcited by 21
  9. 2023Rethinking the Evaluation Protocol of Domain Generalizationcited by 20
  10. 2024A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classificationcited by 19
  11. 2023ERM++: An Improved Baseline for Domain Generalizationcited by 18
  12. 2022You Only Need a Good Embeddings Extractor to Fix Spurious Correlationscited by 17
38 more
  1. 2023The Role of Pre-training Data in Transfer Learningcited by 17
  2. 2023GeoNet: Benchmarking Unsupervised Adaptation across Geographiescited by 17
  3. 2023Learning Sample Difficulty from Pre-trained Models for Reliable Predictioncited by 15
  4. 2023What, How, and When Should Object Detectors Update in Continually Changing Test Domains?cited by 15
  5. 2024Source-Free Domain Adaptation Guided by Vision and Vision-Language Pre-Trainingcited by 15
  6. 2024Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalizationcited by 14
  7. 2022Rethinking the Role of Pre-Trained Networks in Source-Free Domain Adaptationcited by 13
  8. 2022Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistencycited by 11
  9. 2023A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptationcited by 11
  10. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  11. 2023Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learningcited by 9
  12. 2023Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameterscited by 9
  13. 2024PETALface: Parameter Efficient Transfer Learning for Low-resolution Face Recognitioncited by 7
  14. 2024Recasting Generic Pretrained Vision Transformers As Object-Centric Scene Encoders For Manipulation Policiescited by 6
  15. 2024UDA-Bench: Revisiting Common Assumptions in Unsupervised Domain Adaptation Using a Standardized Frameworkcited by 6
  16. 2024De-Confusing Pseudo-Labels in Source-Free Domain Adaptationcited by 5
  17. 2024Rethinking Domain Adaptation and Generalization in the Era of CLIPcited by 5
  18. 2022Addressing Distribution Shift at Test Time in Pre-trained Language Modelscited by 4
  19. 2023Can Pre-trained Networks Detect Familiar Out-of-Distribution Data?cited by 4
  20. 2023Language Semantic Graph Guided Data-Efficient Learningcited by 4
  21. 2023Combining inherent knowledge of vision-language models with unsupervised domain adaptation through strong-weak guidancecited by 4
  22. 2024Open-Source AI-based SE Tools: Opportunities and Challenges of Collaborative Software Learningcited by 4
  23. 2024SETA: Semantic-Aware Token Augmentation for Domain Generalizationcited by 3
  24. 2024Boosting Federated Domain Generalization: Understanding the Role of Advanced Pre-Trained Architecturescited by 3
  25. 2022Okapi: Generalising Better by Making Statistical Matches Matchcited by 2
  26. 2022Reconciling a Centroid-Hypothesis Conflict in Source-Free Domain Adaptationcited by 2
  27. 2023Key Design Choices for Double-Transfer in Source-Free Unsupervised Domain Adaptationcited by 2
  28. 2023Mind the Backbone: Minimizing Backbone Distortion for Robust Object Detectioncited by 2
  29. 2023Incorporating Pre-training Data Matters in Unsupervised Domain Adaptationcited by 2
  30. 2023Benchmarking Low-Shot Robustness to Natural Distribution Shiftscited by 1
  31. 2023Learning for Transductive Threshold Calibration in Open-World Recognitioncited by 1
  32. 2023Rapid Plug-in Defenderscited by 1
  33. 2023Confounder Balancing in Adversarial Domain Adaptation for Pre-Trained Large Models Fine-Tuningcited by 1
  34. 2024Key Design Choices in Source-Free Unsupervised Domain Adaptation: An In-depth Empirical Analysiscited by 1
  35. 2024NC-TTT: A Noise Contrastive Approach for Test-Time Trainingcited by 1
  36. 2024DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalizationcited by 1
  37. 2024Rethinking Pre-Trained Feature Extractor Selection in Multiple Instance Learning for Whole Slide Image Classificationcited by 1
  38. 2024Attention Head Purification: A New Perspective to Harness CLIP for Domain Generalizationcited by 1

50 most cited of 64 All 64 (JSON)