mlsanityData to 7 Oct 2026

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings

arXiv 1909.00512 ↗First posted 2 Sep 2019cs.CLcited by 770

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks. However, just how contextual are the contextualized representations produced by models such as ELMo and BERT? Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations? For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers. This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations. In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.

Key facts. Cited by 770 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 85; 2022: 95; 2023: 100; 2024: 108; 2025: 113; 2026: 269.

770 papers.

Sort:

By area:

  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  3. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  4. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  7. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  8. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  9. 2022A Contrastive Framework for Neural Text Generationcited by 224
  10. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  11. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  12. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
38 more
  1. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  2. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 163
  3. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  4. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  5. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  6. 2022Analyzing Transformers in Embedding Spacecited by 131
  7. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  8. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  9. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  10. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 106
  11. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  12. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  13. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  14. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  15. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  16. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  17. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 70
  18. 2021Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classificationcited by 67
  19. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  20. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  21. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  22. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 58
  23. 2025On the Emergence of Position Bias in Transformerscited by 57
  24. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 57
  25. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  26. 2022Discovering Latent Concepts Learned in BERTcited by 56
  27. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  28. 2023A Study on ReLU and Softmax in Transformercited by 53
  29. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
  30. 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 51
  31. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  32. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  33. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  34. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  35. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  36. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  37. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  38. 2024Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semanticscited by 43

50 most cited of 770 All 770 (JSON)