# Effects of Pre- and Post-Processing on type-based Embeddings in Lexical Semantic Change Detection

arXiv: https://arxiv.org/abs/2101.09368 · HTML: https://mlsanity.com/paper/2101.09368/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 54 references that are in the map.
- First posted on arXiv 22 Jan 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Temporal Analysis of Language through Neural Language Models](https://mlsanity.com/paper/1405.3515/) (2014)
- [Simple and Effective Dimensionality Reduction for Word Embeddings](https://mlsanity.com/paper/1708.03629/) (2017)
- [Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change](https://mlsanity.com/paper/1804.06517/) (2018)
- [Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change](https://mlsanity.com/paper/1906.01688/) (2019)
- [UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection](https://mlsanity.com/paper/2005.00050/) (2020)
- [A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains](https://mlsanity.com/paper/1906.02979/) (2019)
- [RuSemShift: a dataset of historical lexical semantic change in Russian](https://mlsanity.com/paper/2010.06436/) (2020)
- [Challenges for Computational Lexical Semantic Change](https://mlsanity.com/paper/2101.07668/) (2021)
- [Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation](https://mlsanity.com/paper/1809.02094/) (2018)
- [Representing Sentences as Low-Rank Subspaces](https://mlsanity.com/paper/1704.05358/) (2017)
- [UWB at SemEval-2020 Task 1: Lexical Semantic Change Detection](https://mlsanity.com/paper/2012.00004/) (2020)
- [IMS at SemEval-2020 Task 1: How low can you go? Dimensionality in Lexical Semantic Change Detection](https://mlsanity.com/paper/2008.03164/) (2020)
- [OP-IMS @ DIACR-Ita: Back to the Roots: SGNS+OP+CD still rocks Semantic Change Detection](https://mlsanity.com/paper/2011.03258/) (2020)
- [CL-IMS @ DIACR-Ita: Volente o Nolente: BERT does not outperform SGNS on Semantic Change Detection](https://mlsanity.com/paper/2011.07247/) (2020)
- [Second-order Co-occurrence Sensitivity of Skip-Gram with Negative Sampling](https://mlsanity.com/paper/1906.02479/) (2019)
- [Unsupervised Embedding-based Detection of Lexical Semantic Changes](https://mlsanity.com/paper/2005.07979/) (2020)

## Built on by (most cited first, 4 of 4)

- [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022)
- [Lexical Semantic Change Discovery](https://mlsanity.com/paper/2106.03111/) (2021)
- [Graph-based Clustering for Detecting Semantic Change Across Time and Languages](https://mlsanity.com/paper/2402.01025/) (2024)
- [Presence or Absence: Are Unknown Word Usages in Dictionaries?](https://mlsanity.com/paper/2406.00656/) (2024)

## Abstract (arXiv)

Lexical semantic change detection is a new and innovative research field. The optimal fine-tuning of models including pre- and post-processing is largely unclear. We optimize existing models by (i) pre-training on large corpora and refining on diachronic target corpora tackling the notorious small data problem, and (ii) applying post-processing transformations that have been shown to improve performance on synchronic tasks. Our results provide a guide for the application and optimization of lexical semantic change detection models across various learning scenarios.