# Invariance and identifiability issues for word embeddings

arXiv: https://arxiv.org/abs/1911.02656 · HTML: https://mlsanity.com/paper/1911.02656/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2019.

## Built on by (most cited first, 2 of 2)

- [When are Post-hoc Conceptual Explanations Identifiable?](https://mlsanity.com/paper/2206.13872/) (2022)
- [Dynamics of Meaning: Towards the Evaluation of Diachronic Semantic Change in Sinhala](https://mlsanity.com/paper/2609.08609/) (2026)

## Abstract (arXiv)

Word embeddings are commonly obtained as optimizers of a criterion function f of a text corpus, but assessed on word-task performance using a different evaluation function g of the test data. We contend that a possible source of disparity in performance on tasks is the incompatibility between classes of transformations that leave f and g invariant. In particular, word embeddings defined by f are not unique; they are defined only up to a class of transformations to which f is invariant, and this class is larger than the class to which g is invariant. One implication of this is that the apparent superiority of one word embedding over another, as measured by word task performance, may largely be a consequence of the arbitrary elements selected from the respective solution sets. We provide a formal treatment of the above identifiability issue, present some numerical examples, and discuss possible resolutions.