# Open Sesame: Getting Inside BERT's Linguistic Knowledge

arXiv: https://arxiv.org/abs/1906.01698 · HTML: https://mlsanity.com/paper/1906.01698/

## Key facts

- Cited by 120 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2019.

## Built on by (most cited first, 30 of 120)

- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [Are Emergent Abilities in Large Language Models just In-Context Learning?](https://mlsanity.com/paper/2309.01809/) (2023)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [Self-Attention Networks Can Process Bounded Hierarchical Languages](https://mlsanity.com/paper/2105.11115/) (2021)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers](https://mlsanity.com/paper/2203.17247/) (2022)
- [Incorporating Residual and Normalization Layers into Analysis of Masked Language Models](https://mlsanity.com/paper/2109.07152/) (2021)
- [AttentionViz: A Global View of Transformer Attention](https://mlsanity.com/paper/2305.03210/) (2023)
- [Quantifying Context Mixing in Transformers](https://mlsanity.com/paper/2301.12971/) (2023)
- [TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference](https://mlsanity.com/paper/2105.11618/) (2021)
- [The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders](https://mlsanity.com/paper/2109.06939/) (2021)
- [The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color](https://mlsanity.com/paper/2110.08182/) (2021)
- [Word Order Does Matter (And Shuffled Language Models Know It)](https://mlsanity.com/paper/2203.10995/) (2022)
- [Local Interpretations for Explainable Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.11072/) (2021)
- [Does Pretraining for Summarization Require Knowledge Transfer?](https://mlsanity.com/paper/2109.04953/) (2021)
- [Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars](https://mlsanity.com/paper/2312.01429/) (2023)
- [Exploring the Role of BERT Token Representations to Explain Sentence Probing Results](https://mlsanity.com/paper/2104.01477/) (2021)
- [Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing](https://mlsanity.com/paper/2109.10540/) (2021)
- [UniTE: Unified Translation Evaluation](https://mlsanity.com/paper/2204.13346/) (2022)
- [Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention](https://mlsanity.com/paper/2104.12874/) (2021)
- [Visualizing and Understanding Patch Interactions in Vision Transformer](https://mlsanity.com/paper/2203.05922/) (2022)
- [Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers](https://mlsanity.com/paper/2404.16367/) (2024)
- [The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models](https://mlsanity.com/paper/2106.01950/) (2021)
- [STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformer](https://mlsanity.com/paper/2206.04727/) (2022)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech](https://mlsanity.com/paper/2301.11462/) (2023)

## Abstract (arXiv)

How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual representations like BERT perform well on tasks that require sensitivity to linguistic structure. We present here two studies which aim to provide a better understanding of the nature of BERT's representations. The first of these focuses on the identification of structurally-defined elements using diagnostic classifiers, while the second explores BERT's representation of subject-verb agreement and anaphor-antecedent dependencies through a quantitative assessment of self-attention vectors. In both cases, we find that BERT encodes positional information about word tokens well on its lower layers, but switches to a hierarchically-oriented encoding on higher layers. We conclude then that BERT's representations do indeed model linguistically relevant aspects of hierarchical structure, though they do not appear to show the sharp sensitivity to hierarchical structure that is found in human processing of reflexive anaphora.