mlsanityData to 7 Oct 2026

Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changes

arXiv 2407.02820 ↗First posted 3 Jul 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The sense-aware contextualised word embeddings (SCWEs) encode semantic changes of words within the contextualised word embedding (CWE) spaces. Despite the superior performance of SCWEs in contextual/temporal semantic change detection (SCD) benchmarks, it remains unclear as to how the meaning changes are encoded in the embedding space. To study this, we compare pre-trained CWEs and their fine-tuned versions on contextual and temporal semantic change benchmarks under Principal Component Analysis (PCA) and Independent Component Analysis (ICA) transformations. Our experimental results reveal (a) although there exist a smaller number of axes that are specific to semantic changes of words in the pre-trained CWE space, this information gets distributed across all dimensions when fine-tuned, and (b) in contrast to prior work studying the geometry of CWEs, we find that PCA to better represent semantic changes than ICA within the top 10% of axes. These findings encourage the development of more efficient SCD methods with a small number of SCD-aware dimensions. Source code is available at https://github.com/LivNLP/svp-dims .

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

23 of 31 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  5. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  6. 2020SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detectioncited by 105
  7. 2019Evaluating the Underlying Gender Bias in Contextualized Word Embeddingscited by 94
  8. 2014Statistically Significant Detection of Linguistic Changecited by 82
  9. 2018Survey of Computational Approaches to Lexical Semantic Changecited by 60
  10. 2014Temporal Analysis of Language through Neural Language Modelscited by 53
  11. 2021Time Masking for Temporal Language Modelscited by 46
  12. 2020XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualizationcited by 42
11 more
  1. 2017Dynamic Word Embeddings for Evolving Semantic Discoverycited by 38
  2. 2021Explaining and Improving BERT Performance on Lexical Semantic Change Detectioncited by 33
  3. 2020UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detectioncited by 32
  4. 2024A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Changecited by 21
  5. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  6. 2021AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examplescited by 14
  7. 2023Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddingscited by 8
  8. 2024Definition generation for lexical semantic change detectioncited by 7
  9. 2023textitSwap and Predict -- Predicting the Semantic Changes in Words across Corpora by Context Swappingcited by 6
  10. 2024A Semantic Distance Metric Learning approach for Lexical Semantic Change Detectioncited by 5
  11. 2022Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025SCDTour: Embedding Axis Ordering and Merging for Interpretable Semantic Change Detectioncited by 4
  2. 2026Improving Interpretability of Lexical Semantic Change with Neurobiological Featurescited by 1
  3. 2025Tracking Temporal Dynamics of Vector Sets with Gaussian Processcited by 0
  4. 2026Coupled Usage-Sense Processes: Temporal and Attributable Lexical Semantic Changecited by 0