# Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers

arXiv: https://arxiv.org/abs/2109.08406 · HTML: https://mlsanity.com/paper/2109.08406/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives](https://mlsanity.com/paper/1909.01380/) (2019)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [Similarity Analysis of Contextual Word Representation Models](https://mlsanity.com/paper/2005.01172/) (2020)
- [jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models](https://mlsanity.com/paper/2003.02249/) (2020)
- [Residual Networks Behave Like Ensembles of Relatively Shallow Networks](https://mlsanity.com/paper/1605.06431/) (2016)
- [Undivided Attention: Are Intermediate Layers Necessary for BERT?](https://mlsanity.com/paper/2012.11881/) (2020)

## Built on by (most cited first, 30 of 30)

- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [Unified Visual Transformer Compression](https://mlsanity.com/paper/2203.08243/) (2022)
- [Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning](https://mlsanity.com/paper/2504.02922/) (2025)
- [Exploring Representations and Interventions in Time Series Foundation Models](https://mlsanity.com/paper/2409.12915/) (2024)
- [Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning](https://mlsanity.com/paper/2210.10041/) (2022)
- [Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups](https://mlsanity.com/paper/2410.21508/) (2024)
- [Learning to Interpret Weight Differences in Language Models](https://mlsanity.com/paper/2510.05092/) (2025)
- [Representation Potentials of Foundation Models for Multimodal Alignment: A Survey](https://mlsanity.com/paper/2510.05184/) (2025)
- [G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks](https://mlsanity.com/paper/2212.03613/) (2022)
- [Tending Towards Stability: Convergence Challenges in Small Language Models](https://mlsanity.com/paper/2410.11451/) (2024)
- [Collective Model Intelligence Requires Compatible Specialization](https://mlsanity.com/paper/2411.02207/) (2024)
- [CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model](https://mlsanity.com/paper/2310.15477/) (2023)
- [Cross-layer Attention Sharing for Pre-trained Large Language Models](https://mlsanity.com/paper/2408.01890/) (2024)
- [Life after BERT: What do Other Muppets Understand about Language?](https://mlsanity.com/paper/2205.10696/) (2022)
- [Spectral Insights into Data-Oblivious Critical Layers in Large Language Models](https://mlsanity.com/paper/2506.00382/) (2025)
- [ADEPT: Adaptive Dynamic Early-Exit Process for Transformers](https://mlsanity.com/paper/2601.03700/) (2026)
- [VendorLink: An NLP approach for Identifying & Linking Vendor Migrants & Potential Aliases on Darknet Markets](https://mlsanity.com/paper/2305.02763/) (2023)
- [What Happens During Finetuning of Vision Transformers: An Invariance Based Investigation](https://mlsanity.com/paper/2307.06006/) (2023)
- [Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models](https://mlsanity.com/paper/2601.03542/) (2026)
- [A Single Linear Layer Yields Task-Adapted Low-Rank Matrices](https://mlsanity.com/paper/2403.14946/) (2024)
- [Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility](https://mlsanity.com/paper/2608.03930/) (2026)
- [Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models](https://mlsanity.com/paper/2308.08774/) (2023)
- [Where does In-context Translation Happen in Large Language Models](https://mlsanity.com/paper/2403.04510/) (2024)
- [A Representation Level Analysis of NMT Model Robustness to Grammatical Errors](https://mlsanity.com/paper/2505.21224/) (2025)
- [Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language](https://mlsanity.com/paper/2605.15607/) (2026)
- [Perturbed examples reveal invariances shared by language models](https://mlsanity.com/paper/2311.04166/) (2023)
- [Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models](https://mlsanity.com/paper/2602.02304/) (2026)
- [Continued Pretraining of FinBERT on Finnish Histopathological Reports: Train-Time Signals and Proxy Downstream Correlations](https://mlsanity.com/paper/2604.14815/) (2026)
- [The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models](https://mlsanity.com/paper/2609.38205/) (2026)
- [Architecture-Dependent Fusion Pathways in MLLMs](https://mlsanity.com/paper/2610.03289/) (2026)

## Abstract (arXiv)

Despite the success of fine-tuning pretrained language encoders like BERT for downstream natural language understanding (NLU) tasks, it is still poorly understood how neural networks change after fine-tuning. In this work, we use centered kernel alignment (CKA), a method for comparing learned representations, to measure the similarity of representations in task-tuned models across layers. In experiments across twelve NLU tasks, we discover a consistent block diagonal structure in the similarity of representations within fine-tuned RoBERTa and ALBERT models, with strong similarity within clusters of earlier and later layers, but not between them. The similarity of later layer representations implies that later layers only marginally contribute to task performance, and we verify in experiments that the top few layers of fine-tuned Transformers can be discarded without hurting performance, even with no further tuning.