mlsanityData to 7 Oct 2026

Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses

arXiv 2104.14694 ↗First posted 29 Apr 2021cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analysing this knowledge in LMs specifically trained for different languages (English, French, Spanish and Greek) and in multilingual BERT. We perform our analysis on datasets carefully designed to reflect different sense distributions, and control for parameters that are highly correlated with polysemy such as frequency and grammatical category. We demonstrate that BERT-derived representations reflect words' polysemy level and their partitionability into senses. Polysemy-related information is more clearly present in English BERT embeddings, but models in other languages also manage to establish relevant distinctions between words at different polysemy levels. Our results contribute to a better understanding of the knowledge encoded in contextualised representations and open up new avenues for multilingual lexical semantics research.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 33 of its 70 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

33 of 70 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  6. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  10. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  11. 2019Designing and Interpreting Probes with Control Taskscited by 608
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
21 more
  1. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  2. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  3. 2019Revealing the Dark Secrets of BERTcited by 300
  4. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  5. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  6. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  7. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  8. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  9. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  10. 2020SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detectioncited by 105
  11. 2020Analysing Lexical Semantic Change with Contextualised Word Representationscited by 92
  12. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  13. 2018From Word to Sense Embeddings: A Survey on Vector Representations of Meaningcited by 49
  14. 2020GREEK-BERT: The Greeks visiting Sesame Streetcited by 37
  15. 2015Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Spacecited by 35
  16. 2019Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Changecited by 32
  17. 2020Topology of Word Embeddings: Singularities Reflect Polysemycited by 15
  18. 2019Putting words in context: LSTM language models and lexical ambiguitycited by 7
  19. 2019Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutescited by 5
  20. 2020Don't Neglect the Obvious: On the Role of Unambiguous Words in Word Sense Disambiguationcited by 4
  21. 2020Speakers Fill Lexical Semantic Gaps with Contextcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 6; 2023: 11; 2024: 9; 2025: 6; 2026: 2.

39 papers.

Sort:

  1. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  2. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  3. 2023A Survey on Contextualised Semantic Shift Detectioncited by 21
  4. 2023Evaluating Biased Attitude Associations of Language Models in an Intersectional Contextcited by 20
  5. 2021MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Modelscited by 18
  6. 2021LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyondcited by 17
  7. 2023Do Language Models' Words Refer?cited by 17
  8. 2022Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representationscited by 14
  9. 2025LLMs as a synthesis between symbolic and distributed approaches to languagecited by 13
  10. 2024Word-specific tonal realizations in Mandarincited by 9
  11. 2022Transparency Helps Reveal When Language Models Learn Meaningcited by 8
  12. 2023Representation Of Lexical Stylistic Features In Language Models' Embedding Spacecited by 8
27 more
  1. 2024Analyzing Semantic Change through Lexical Replacementscited by 7
  2. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  3. 2024The Impact of Word Splitting on the Semantic Content of Contextualized Word Representationscited by 6
  4. 2023Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Informationcited by 5
  5. 2024Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysiscited by 5
  6. 2021Patterns of Lexical Ambiguity in Contextualised Language Modelscited by 4
  7. 2023ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Modelscited by 4
  8. 2024Adjusting Interpretable Dimensions in Embedding Space with Human Judgmentscited by 4
  9. 2025Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMscited by 4
  10. 2023Word sense extensioncited by 3
  11. 2024Tracing the Development of the Virtual Particle Concept Using Semantic Change Detectioncited by 3
  12. 2025Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguitycited by 3
  13. 2022Reconstruction Probingcited by 2
  14. 2023Construction Grammar and Language Modelscited by 2
  15. 2025Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddingscited by 2
  16. 2021Measuring Context-Word Biases in Lexical Semantic Datasetscited by 1
  17. 2023Exploring Category Structure with Contextual Language Models and Lexical Semantic Networkscited by 1
  18. 2023Attention is Not Always What You Need: Towards Efficient Classification of Domain-Specific Textcited by 1
  19. 2024A Systematic Survey of Natural Language Processing for the Greek Languagecited by 1
  20. 2024BERT's Conceptual Cartography: Mapping the Landscapes of Meaningcited by 1
  21. 2024Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspectivecited by 1
  22. 2022Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddingscited by 0
  23. 2022Measure More, Question More: Experimental Studies on Transformer-based Language Models and Complement Coercioncited by 0
  24. 2023Systematic word meta-sense extensioncited by 0
  25. 2025The cell as a token: high-dimensional geometry in language models and cell embeddingscited by 0
  26. 2026Chatbots Output Meaningful (but Problematic) Languagecited by 0
  27. 2026Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropycited by 0