# Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representations

arXiv: https://arxiv.org/abs/2109.05958 · HTML: https://mlsanity.com/paper/2109.05958/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives](https://mlsanity.com/paper/1909.01380/) (2019)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals](https://mlsanity.com/paper/1911.10422/) (2019)
- [BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?](https://mlsanity.com/paper/2105.04949/) (2021)
- [Probing BERT in Hyperbolic Spaces](https://mlsanity.com/paper/2104.03869/) (2021)
- [On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers](https://mlsanity.com/paper/2010.02616/) (2020)
- [What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models](https://mlsanity.com/paper/2004.06499/) (2020)
- [Exploring the Role of BERT Token Representations to Explain Sentence Probing Results](https://mlsanity.com/paper/2104.01477/) (2021)
- [What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties](https://mlsanity.com/paper/2009.12862/) (2020)
- [A Cross-Task Analysis of Text Span Representations](https://mlsanity.com/paper/2006.03866/) (2020)
- [Analyzing analytical methods: The case of phonology in neural models of spoken language](https://mlsanity.com/paper/2004.07070/) (2020)
- [Inspecting the concept knowledge graph encoded by modern language models](https://mlsanity.com/paper/2105.13471/) (2021)
- [A Matter of Framing: The Impact of Linguistic Formalism on Probing Results](https://mlsanity.com/paper/2004.14999/) (2020)

## Built on by (most cited first, 16 of 16)

- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://mlsanity.com/paper/2309.03883/) (2023)
- [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://mlsanity.com/paper/2305.00586/) (2023)
- [GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers](https://mlsanity.com/paper/2205.03286/) (2022)
- [Quantifying Context Mixing in Transformers](https://mlsanity.com/paper/2301.12971/) (2023)
- [Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages](https://mlsanity.com/paper/2203.14139/) (2022)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [A Meta-Learning Perspective on Transformers for Causal Language Modeling](https://mlsanity.com/paper/2310.05884/) (2023)
- [Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation](https://mlsanity.com/paper/2210.13236/) (2022)
- [Sort by Structure: Language Model Ranking as Dependency Probing](https://mlsanity.com/paper/2206.04935/) (2022)
- [Norm of Mean Contextualized Embeddings Determines their Variance](https://mlsanity.com/paper/2409.11253/) (2024)
- [How Language Models Prioritize Contextual Grammatical Cues?](https://mlsanity.com/paper/2410.03447/) (2024)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [AuthorMix: Modular Authorship Style Transfer via Layer-wise Adapter Mixing](https://mlsanity.com/paper/2603.23069/) (2026)
- [Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs](https://mlsanity.com/paper/2605.12714/) (2026)
- [Towards Probing Contact Center Large Language Models](https://mlsanity.com/paper/2312.15922/) (2023)
- [Categorize Early, Integrate Late: Divergent Processing Strategies in Automatic Speech Recognition](https://mlsanity.com/paper/2601.06972/) (2026)

## Abstract (arXiv)

Most of the recent works on probing representations have focused on BERT, with the presumption that the findings might be similar to the other models. In this work, we extend the probing studies to two other models in the family, namely ELECTRA and XLNet, showing that variations in the pre-training objectives or architectural choices can result in different behaviors in encoding linguistic information in the representations. Most notably, we observe that ELECTRA tends to encode linguistic knowledge in the deeper layers, whereas XLNet instead concentrates that in the earlier layers. Also, the former model undergoes a slight change during fine-tuning, whereas the latter experiences significant adjustments. Moreover, we show that drawing conclusions based on the weight mixing evaluation strategy -- which is widely used in the context of layer-wise probing -- can be misleading given the norm disparity of the representations across different layers. Instead, we adopt an alternative information-theoretic probing with minimum description length, which has recently been proven to provide more reliable and informative results.