mlsanityData to 14 Oct 2026

Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy

arXiv 2106.13553 ↗First posted 25 Jun 2021cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

This paper presents a multilingual study of word meaning representations in context. We assess the ability of both static and contextualized models to adequately represent different lexical-semantic relations, such as homonymy and synonymy. To do so, we created a new multilingual dataset that allows us to perform a controlled evaluation of several factors such as the impact of the surrounding context or the overlap between words, conveying the same or different senses. A systematic assessment on four scenarios shows that the best monolingual models based on Transformers can adequately disambiguate homonyms in context. However, as they rely heavily on context, these models fail at representing words with different senses when occurring in similar sentences. Experiments are performed in Galician, Portuguese, English, and Spanish, and both the dataset (with more than 3,000 evaluation items) and new models are freely released with this study.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 24 of its 69 references that are in the map. First posted on arXiv 25 Jun 2021.

Builds on

24 of 69 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2018Deep contextualized word representationscited by 2,290
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  11. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  12. 2019Visualizing and Measuring the Geometry of BERTcited by 216
12 more
  1. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  2. 2019Adaptation of Deep Bidirectional Multilingual Transformers for Russian Languagecited by 109
  3. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 70
  4. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 56
  5. 2020Word meaning in minds and machinescited by 51
  6. 2020XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualizationcited by 42
  7. 2020Assessing Phrasal Representation and Composition in Transformerscited by 41
  8. 2020Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledgecited by 18
  9. 2019UDPipe at SIGMORPHON 2019: Contextualized Embeddings, Regularization with Morphological Categories, Corpora Mergingcited by 15
  10. 2017Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Taskscited by 13
  11. 2019CoSimLex: A Resource for Evaluating Graded Word Similarity in Contextcited by 13
  12. 2019Putting words in context: LSTM language models and lexical ambiguitycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 3; 2025: 3; 2026: 2.

9 papers.

  1. 2024Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysiscited by 5
  2. 2022A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training timecited by 3
  3. 2024Open Generative Large Language Models for Galiciancited by 3
  4. 2025Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguitycited by 3
  5. 2025FinchGPT: a Transformer based language model for birdsong analysiscited by 2
  6. 2024Investigating Idiomaticity in Word Representationscited by 1
  7. 2026Relation Geometry in Semantic Space of Language Modelscited by 1
  8. 2025Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clusteringcited by 0
  9. 2026Beyond Cosine Similarity: Taming Semantic Drift and Antonym Intrusion in a 15-Million Node Turkish Synonym Graphcited by 0