What Happens During Finetuning of Vision Transformers: An Invariance Based Investigation
Not on the map. Why?
Abstract (arXiv)
The pretrain-finetune paradigm usually improves downstream performance over training a model from scratch on the same task, becoming commonplace across many areas of machine learning. While pretraining is empirically observed to be beneficial for a range of tasks, there is not a clear understanding yet of the reasons for this effect. In this work, we examine the relationship between pretrained vision transformers and the corresponding finetuned versions on several benchmark datasets and tasks. We present new metrics that specifically investigate the degree to which invariances learned by a pretrained model are retained or forgotten during finetuning. Using these metrics, we present a suite of empirical findings, including that pretraining induces transferable invariances in shallow layers and that invariances from deeper pretrained layers are compressed towards shallower layers during finetuning. Together, these findings contribute to understanding some of the reasons for the successes of pretrained models and the changes that a pretrained model undergoes when finetuned on a downstream task.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 12 Jul 2023.
Builds on
29 of 38 references.
References · 29
- 2017Attention Is All You Needcited by 59,932
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
- 2013Visualizing and Understanding Convolutional Networkscited by 2,278
- 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
- 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
17 more
- 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
- 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
- 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
- 2018Do Better ImageNet Models Transfer Better?cited by 560
- 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
- 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
- 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
- 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
- 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
- 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
- 2016What makes ImageNet good for transfer learning?cited by 192
- 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
- 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
- 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 74
- 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
- 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 29
- 2022Measuring Representational Robustness of Neural Networks Through Shared Invariancescited by 11
Built on by
Citing papers per year: 2024: 2; 2025: 0; 2026: 1.
3 papers.