# Linguistic Interpretability of Transformer-based Language Models: a systematic review

arXiv: https://arxiv.org/abs/2504.08001 · HTML: https://mlsanity.com/paper/2504.08001/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 95 of its 188 references that are in the map.
- First posted on arXiv 9 Apr 2025.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 30 of 95)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [Self-Attention Attribution: Interpreting Information Interactions Inside Transformer](https://mlsanity.com/paper/2004.11207/) (2020)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [When Do You Need Billions of Words of Pretraining Data?](https://mlsanity.com/paper/2011.04946/) (2020)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Do Attention Heads in BERT Track Syntactic Dependencies?](https://mlsanity.com/paper/1911.12246/) (2019)

## Built on by (most cited first, 7 of 7)

- [Vision Function Layer in Multimodal LLMs](https://mlsanity.com/paper/2509.24791/) (2025)
- [A Statistical Physics of Language Model Reasoning](https://mlsanity.com/paper/2506.04374/) (2025)
- [Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks](https://mlsanity.com/paper/2601.03448/) (2026)
- [Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect](https://mlsanity.com/paper/2509.12065/) (2025)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Probing the Embedding Space of Transformers via Minimal Token Perturbations](https://mlsanity.com/paper/2506.18011/) (2025)
- [Technical Manual for a Toolkit for Measuring Contextual Individuation in Transformer Language Models](https://mlsanity.com/paper/2609.05333/) (2026)

## Abstract (arXiv)

Language models based on the Transformer architecture achieve excellent results in many language-related tasks, such as text classification or sentiment analysis. However, despite the architecture of these models being well-defined, little is known about how their internal computations help them achieve their results. This renders these models, as of today, a type of 'black box' systems. There is, however, a line of research -- 'interpretability' -- aiming to learn how information is encoded inside these models. More specifically, there is work dedicated to studying whether Transformer-based models possess knowledge of linguistic phenomena similar to human speakers -- an area we call 'linguistic interpretability' of these models. In this survey we present a comprehensive analysis of 160 research works, spread across multiple languages and models -- including multilingual ones -- that attempt to discover linguistic information from the perspective of several traditional Linguistics disciplines: Syntax, Morphology, Lexico-Semantics and Discourse. Our survey fills a gap in the existing interpretability literature, which either not focus on linguistic knowledge in these models or present some limitations -- e.g. only studying English-based models. Our survey also focuses on Pre-trained Language Models not further specialized for a downstream task, with an emphasis on works that use interpretability techniques that explore models' internal representations.