# Quantifying the Contextualization of Word Representations with Semantic Class Probing

arXiv: https://arxiv.org/abs/2004.12198 · HTML: https://mlsanity.com/paper/2004.12198/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2020.

## Built on by (most cited first, 18 of 18)

- [Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages](https://mlsanity.com/paper/2203.14139/) (2022)
- [Visual Comparison of Language Model Adaptation](https://mlsanity.com/paper/2208.08176/) (2022)
- [DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generation](https://mlsanity.com/paper/2209.01339/) (2022)
- [Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords](https://mlsanity.com/paper/2109.11491/) (2021)
- [LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework](https://mlsanity.com/paper/2112.07522/) (2021)
- [Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities](https://mlsanity.com/paper/2212.10017/) (2022)
- [SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings](https://mlsanity.com/paper/2301.04704/) (2023)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations](https://mlsanity.com/paper/2305.14555/) (2023)
- [GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models](https://mlsanity.com/paper/2410.06154/) (2024)
- [What do Large Language Models Learn beyond Language?](https://mlsanity.com/paper/2210.12302/) (2022)
- [Where exactly does contextualization in a PLM happen?](https://mlsanity.com/paper/2312.06514/) (2023)
- [Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?](https://mlsanity.com/paper/2405.07099/) (2024)
- [On Isotropy Calibration of Transformers](https://mlsanity.com/paper/2109.13304/) (2021)
- [Probing of Quantitative Values in Abstractive Summarization Models](https://mlsanity.com/paper/2210.00667/) (2022)
- [Probing Context Localization of Polysemous Words in Pre-trained Language Model Sub-Layers](https://mlsanity.com/paper/2409.14097/) (2024)
- [Towards Explainable Job Title Matching: Leveraging Semantic Textual Relatedness and Knowledge Graphs](https://mlsanity.com/paper/2509.09522/) (2025)
- [A graph-based analysis of semantic types and coercion in contextualized word embeddings](https://mlsanity.com/paper/2605.23710/) (2026)

## Abstract (arXiv)

Pretrained language models have achieved a new state of the art on many NLP tasks, but there are still many open questions about how and why they work so well. We investigate the contextualization of words in BERT. We quantify the amount of contextualization, i.e., how well words are interpreted in context, by studying the extent to which semantic classes of a word can be inferred from its contextualized embeddings. Quantifying contextualization helps in understanding and utilizing pretrained language models. We show that top layer representations achieve high accuracy inferring semantic classes; that the strongest contextualization effects occur in the lower layers; that local context is mostly sufficient for semantic class inference; and that top layer representations are more task-specific after finetuning while lower layer representations are more transferable. Finetuning uncovers task related features, but pretrained knowledge is still largely preserved.