# Similarity of Pre-trained and Fine-tuned Representations

arXiv: https://arxiv.org/abs/2207.09225 · HTML: https://mlsanity.com/paper/2207.09225/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 11 references that are in the map.
- First posted on arXiv 19 Jul 2022.

## Builds on (most cited first, 9 of 9)

- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [Why Do Better Loss Functions Lead to Less Transferable Features?](https://mlsanity.com/paper/2010.16402/) (2020)
- [Theory of Deep Learning III: explaining the non-overfitting puzzle](https://mlsanity.com/paper/1801.00173/) (2017)
- [Duality Diagram Similarity: a generic framework for initialization selection in task transfer learning](https://mlsanity.com/paper/2008.02107/) (2020)
- [Exploring the Similarity of Representations in Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2105.05757/) (2021)

## Abstract (arXiv)

In transfer learning, only the last part of the networks - the so-called head - is often fine-tuned. Representation similarity analysis shows that the most significant change still occurs in the head even if all weights are updatable. However, recent results from few-shot learning have shown that representation change in the early layers, which are mostly convolutional, is beneficial, especially in the case of cross-domain adaption. In our paper, we find out whether that also holds true for transfer learning. In addition, we analyze the change of representation in transfer learning, both during pre-training and fine-tuning, and find out that pre-trained structure is unlearned if not usable.