# Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks

arXiv: https://arxiv.org/abs/2102.00290 · HTML: https://mlsanity.com/paper/2102.00290/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 30 Jan 2021.

## Builds on (most cited first, 22 of 22)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection](https://mlsanity.com/paper/2007.11464/) (2020)
- [Diachronic word embeddings and semantic shifts: a survey](https://mlsanity.com/paper/1806.03537/) (2018)
- [Statistically Significant Detection of Linguistic Change](https://mlsanity.com/paper/1411.3315/) (2014)
- [Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion](https://mlsanity.com/paper/1804.07745/) (2018)
- [Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change](https://mlsanity.com/paper/1606.02821/) (2016)
- [Dynamic Word Embeddings](https://mlsanity.com/paper/1702.08359/) (2017)
- [Dynamic Word Embeddings for Evolving Semantic Discovery](https://mlsanity.com/paper/1703.00607/) (2017)
- [A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains](https://mlsanity.com/paper/1906.02979/) (2019)
- [Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework](https://mlsanity.com/paper/1910.04708/) (2019)
- [The Global Anchor Method for Quantifying Linguistic Shifts and Domain Adaptation](https://mlsanity.com/paper/1812.10382/) (2018)
- [Aligning Vector-spaces with Noisy Supervised Lexicons](https://mlsanity.com/paper/1903.10238/) (2019)
- [Updating Pre-trained Word Vectors and Text Classifiers using Monolingual Alignment](https://mlsanity.com/paper/1910.06241/) (2019)
- [Temporal dynamics of semantic relations in word embeddings: an application to predicting armed conflict participants](https://mlsanity.com/paper/1707.08660/) (2017)

## Built on by (most cited first, 1 of 1)

- [Adversarial Machine Learning for Social Good: Reframing the Adversary as an Ally](https://mlsanity.com/paper/2310.03614/) (2023)

## Abstract (arXiv)

The use of language is subject to variation over time as well as across social groups and knowledge domains, leading to differences even in the monolingual scenario. Such variation in word usage is often called lexical semantic change (LSC). The goal of LSC is to characterize and quantify language variations with respect to word meaning, to measure how distinct two language sources are (that is, people or language models). Because there is hardly any data available for such a task, most solutions involve unsupervised methods to align two embeddings and predict semantic change with respect to a distance measure. To that end, we propose a self-supervised approach to model lexical semantic change by generating training samples by introducing perturbations of word vectors in the input corpora. We show that our method can be used for the detection of semantic change with any alignment method. Furthermore, it can be used to choose the landmark words to use in alignment and can lead to substantial improvements over the existing techniques for alignment. We illustrate the utility of our techniques using experimental results on three different datasets, involving words with the same or different meanings. Our methods not only provide significant improvements but also can lead to novel findings for the LSC problem.