# When Word Embeddings Become Endangered

arXiv: https://arxiv.org/abs/2103.13275 · HTML: https://mlsanity.com/paper/2103.13275/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://mlsanity.com/paper/1802.05368/) (2018)
- [Simple and Effective Dimensionality Reduction for Word Embeddings](https://mlsanity.com/paper/1708.03629/) (2017)
- [A Large-Scale Comparison of Historical Text Normalization Systems](https://mlsanity.com/paper/1904.02036/) (2019)
- [Simulated Multiple Reference Training Improves Low-Resource Machine Translation](https://mlsanity.com/paper/2004.14524/) (2020)
- [Learning Crosslingual Word Embeddings without Bilingual Corpora](https://mlsanity.com/paper/1606.09403/) (2016)
- [Let's FACE it. Finnish Poetry Generation with Aesthetics and Framing](https://mlsanity.com/paper/1910.13946/) (2019)
- [Improving Candidate Generation for Low-resource Cross-lingual Entity Linking](https://mlsanity.com/paper/2003.01343/) (2020)
- [FST Morphology for the Endangered Skolt Sami Language](https://mlsanity.com/paper/2004.04803/) (2020)
- [Ve'rdd. Narrowing the Gap between Paper Dictionaries, Low-Resource NLP and Community Involvement](https://mlsanity.com/paper/2012.02578/) (2020)

## Built on by (most cited first, 6 of 6)

- [The first neural machine translation system for the Erzya language](https://mlsanity.com/paper/2209.09368/) (2022)
- [Evaluating Transferability of BERT Models on Uralic Languages](https://mlsanity.com/paper/2109.06327/) (2021)
- [Never guess what I heard... Rumor Detection in Finnish News: a Dataset and a Baseline](https://mlsanity.com/paper/2106.03389/) (2021)
- [Apurin~a Universal Dependencies Treebank](https://mlsanity.com/paper/2106.03391/) (2021)
- [Video Games as a Corpus: Sentiment Analysis using Fallout New Vegas Dialog](https://mlsanity.com/paper/2212.02168/) (2022)
- [Sentiment Analysis Using Aligned Word Embeddings for Uralic Languages](https://mlsanity.com/paper/2305.15380/) (2023)

## Abstract (arXiv)

Big languages such as English and Finnish have many natural language processing (NLP) resources and models, but this is not the case for low-resourced and endangered languages as such resources are so scarce despite the great advantages they would provide for the language communities. The most common types of resources available for low-resourced and endangered languages are translation dictionaries and universal dependencies. In this paper, we present a method for constructing word embeddings for endangered languages using existing word embeddings of different resource-rich languages and the translation dictionaries of resource-poor languages. Thereafter, the embeddings are fine-tuned using the sentences in the universal dependencies and aligned to match the semantic spaces of the big languages; resulting in cross-lingual embeddings. The endangered languages we work with here are Erzya, Moksha, Komi-Zyrian and Skolt Sami. Furthermore, we build a universal sentiment analysis model for all the languages that are part of this study, whether endangered or not, by utilizing cross-lingual word embeddings. The evaluation conducted shows that our word embeddings for endangered languages are well-aligned with the resource-rich languages, and they are suitable for training task-specific models as demonstrated by our sentiment analysis model which achieved a high accuracy. All our cross-lingual word embeddings and the sentiment analysis model have been released openly via an easy-to-use Python library.