mlsanityData to 7 Oct 2026

Explaining and Improving BERT Performance on Lexical Semantic Change Detection

arXiv 2103.07259 ↗First posted 12 Mar 2021cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Type- and token-based embedding architectures are still competing in lexical semantic change detection. The recent success of type-based models in SemEval-2020 Task 1 has raised the question why the success of token-based models on a variety of other NLP tasks does not translate to our field. We investigate the influence of a range of variables on clusterings of BERT vectors and show that its low performance is largely due to orthographic information on the target word, which is encoded even in the higher layers of BERT representations. By reducing the influence of orthography we considerably improve BERT's performance.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 15 of its 33 references that are in the map. First posted on arXiv 12 Mar 2021.

Builds on

15 of 33 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Deep contextualized word representationscited by 2,285
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  4. 2020SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detectioncited by 105
  5. 2020Analysing Lexical Semantic Change with Contextualised Word Representationscited by 92
  6. 2018Diachronic word embeddings and semantic shifts: a surveycited by 90
  7. 2018Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Changecited by 35
  8. 2019Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Changecited by 32
  9. 2020UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detectioncited by 32
  10. 2019A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domainscited by 31
  11. 2020Capturing Evolution in Word Usage: Just Add More Clusters?cited by 22
  12. 2021Challenges for Computational Lexical Semantic Changecited by 15
3 more
  1. 2017Dialectometric analysis of language variation in Twittercited by 14
  2. 2020UWB at SemEval-2020 Task 1: Lexical Semantic Change Detectioncited by 7
  3. 2020OP-IMS @ DIACR-Ita: Back to the Roots: SGNS+OP+CD still rocks Semantic Change Detectioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 7; 2024: 13; 2025: 2; 2026: 2.

33 papers.

Sort:

By area:

  1. 2021Time Masking for Temporal Language Modelscited by 46
  2. 2022Temporal Attention for Language Modelscited by 21
  3. 2023A Survey on Contextualised Semantic Shift Detectioncited by 21
  4. 2024A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Changecited by 21
  5. 2022Slangvolution: A Causal Analysis of Semantic Change and Frequency Dynamics in Slangcited by 14
  6. 2021Lexical Semantic Change Discoverycited by 13
  7. 2022LSCDiscovery: A shared task on semantic change discovery and detection in Spanishcited by 11
  8. 2024Graph-based Clustering for Detecting Semantic Change Across Time and Languagescited by 10
  9. 2023Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddingscited by 8
  10. 2024(Chat)GPT v BERT: Dawn of Justice for Semantic Change Detectioncited by 7
  11. 2024Analyzing Semantic Change through Lexical Replacementscited by 7
  12. 2024Using Synchronic Definitions and Semantic Relations to Classify Semantic Change Typescited by 7
21 more
  1. 2023textitSwap and Predict -- Predicting the Semantic Changes in Words across Corpora by Context Swappingcited by 6
  2. 2024The Impact of Word Splitting on the Semantic Content of Contextualized Word Representationscited by 6
  3. 2024A Semantic Distance Metric Learning approach for Lexical Semantic Change Detectioncited by 5
  4. 2022A Greek Parliament Proceedings Dataset for Computational Linguistics and Political Analysiscited by 4
  5. 2024Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changescited by 4
  6. 2025SCDTour: Embedding Axis Ordering and Merging for Interpretable Semantic Change Detectioncited by 4
  7. 2024Detection of Non-recorded Word Senses in English and Swedishcited by 3
  8. 2024Tracing the Development of the Virtual Particle Concept Using Semantic Change Detectioncited by 3
  9. 2025Analyzing Continuous Semantic Shifts with Diachronic Word Similarity Matricescited by 3
  10. 2022BOS at LSCDiscovery: Lexical Substitution for Interpretable Lexical Semantic Change Detectioncited by 2
  11. 2022Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiCcited by 2
  12. 2023Computational modeling of semantic changecited by 2
  13. 2024The LSCD Benchmark: a Testbed for Diachronic Word Meaning Taskscited by 2
  14. 2023Characterizing and Measuring Linguistic Dataset Driftcited by 1
  15. 2023Mindful Explanations: Prevalence and Impact of Mind Attribution in XAI Researchcited by 1
  16. 2024Tracking Semantic Change in Slovene: A Novel Dataset and Optimal Transport-Based Distancecited by 1
  17. 2026Improving Interpretability of Lexical Semantic Change with Neurobiological Featurescited by 1
  18. 2021Text analysis and deep learning: A network approachcited by 0
  19. 2023CCAE: A Corpus of Chinese-based Asian Englishescited by 0
  20. 2024Deep-change at AXOLOTL-24: Orchestrating WSD and WSI Models for Semantic Change Modelingcited by 0
  21. 2026Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detectioncited by 0