# Training Cross-Lingual embeddings for Setswana and Sepedi

arXiv: https://arxiv.org/abs/2111.06230 · HTML: https://mlsanity.com/paper/2111.06230/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 11 Nov 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://mlsanity.com/paper/2010.02353/) (2020)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Improving zero-shot learning by mitigating the hubness problem](https://mlsanity.com/paper/1412.6568/) (2014)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)
- [Improving short text classification through global augmentation methods](https://mlsanity.com/paper/1907.03752/) (2019)
- [Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yor`ub'a and Twi](https://mlsanity.com/paper/1912.02481/) (2019)
- [A Survey of Word Embeddings Evaluation Methods](https://mlsanity.com/paper/1801.09536/) (2018)
- [A Focus on Neural Machine Translation for African Languages](https://mlsanity.com/paper/1906.05685/) (2019)
- [Investigating an approach for low resource language dataset creation, curation and classification: Setswana and Sepedi](https://mlsanity.com/paper/2003.04986/) (2020)
- [Benchmarking Neural Machine Translation for Southern African Languages](https://mlsanity.com/paper/1906.10511/) (2019)
- [Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study](https://mlsanity.com/paper/2106.04995/) (2021)

## Built on by (most cited first, 5 of 5)

- [TIGQA:An Expert Annotated Question Answering Dataset in Tigrinya](https://mlsanity.com/paper/2404.17194/) (2024)
- [XAI in Computational Linguistics: Understanding Political Leanings in the Slovenian Parliament](https://mlsanity.com/paper/2305.04631/) (2023)
- [Detection of depression on social networks using transformers and ensembles](https://mlsanity.com/paper/2305.05325/) (2023)
- [Vulgar Remarks Detection in Chittagonian Dialect of Bangla](https://mlsanity.com/paper/2308.15448/) (2023)
- [Language-Independent Sentiment Labelling with Distant Supervision: A Case Study for English, Sepedi and Setswana](https://mlsanity.com/paper/2511.19818/) (2025)

## Abstract (arXiv)

African languages still lag in the advances of Natural Language Processing techniques, one reason being the lack of representative data, having a technique that can transfer information between languages can help mitigate against the lack of data problem. This paper trains Setswana and Sepedi monolingual word vectors and uses VecMap to create cross-lingual embeddings for Setswana-Sepedi in order to do a cross-lingual transfer. Word embeddings are word vectors that represent words as continuous floating numbers where semantically similar words are mapped to nearby points in n-dimensional space. The idea of word embeddings is based on the distribution hypothesis that states, semantically similar words are distributed in similar contexts (Harris, 1954). Cross-lingual embeddings leverages monolingual embeddings by learning a shared vector space for two separately trained monolingual vectors such that words with similar meaning are represented by similar vectors. In this paper, we investigate cross-lingual embeddings for Setswana-Sepedi monolingual word vector. We use the unsupervised cross lingual embeddings in VecMap to train the Setswana-Sepedi cross-language word embeddings. We evaluate the quality of the Setswana-Sepedi cross-lingual word representation using a semantic evaluation task. For the semantic similarity task, we translated the WordSim and SimLex tasks into Setswana and Sepedi. We release this dataset as part of this work for other researchers. We evaluate the intrinsic quality of the embeddings to determine if there is improvement in the semantic representation of the word embeddings.