# UzBERT: pretraining a BERT model for Uzbek

arXiv: https://arxiv.org/abs/2108.09814 · HTML: https://mlsanity.com/paper/2108.09814/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 22 Aug 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [Polyglot: Distributed Word Representations for Multilingual NLP](https://mlsanity.com/paper/1307.1662/) (2013)
- [KR-BERT: A Small-Scale Korean-Specific Language Model](https://mlsanity.com/paper/2008.03979/) (2020)
- [Cross-Lingual Word Embeddings for Turkic Languages](https://mlsanity.com/paper/2005.08340/) (2020)
- [Uzbek Cyrillic-Latin-Cyrillic Machine Transliteration](https://mlsanity.com/paper/2101.05162/) (2021)
- [Development of Word Embeddings for Uzbek Language](https://mlsanity.com/paper/2009.14384/) (2020)

## Built on by (most cited first, 5 of 5)

- [SimRelUz: Similarity and Relatedness scores as a Semantic Evaluation dataset for Uzbek language](https://mlsanity.com/paper/2205.06072/) (2022)
- [A machine transliteration tool between Uzbek alphabets](https://mlsanity.com/paper/2205.09578/) (2022)
- [Recent Advancements and Challenges of Turkic Central Asian Language Processing](https://mlsanity.com/paper/2407.05006/) (2024)
- [Uzbek Sentiment Analysis based on local Restaurant Reviews](https://mlsanity.com/paper/2205.15930/) (2022)
- [Creating a morphological and syntactic tagged corpus for the Uzbek language](https://mlsanity.com/paper/2210.15234/) (2022)

## Abstract (arXiv)

Pretrained language models based on the Transformer architecture have achieved state-of-the-art results in various natural language processing tasks such as part-of-speech tagging, named entity recognition, and question answering. However, no such monolingual model for the Uzbek language is publicly available. In this paper, we introduce UzBERT, a pretrained Uzbek language model based on the BERT architecture. Our model greatly outperforms multilingual BERT on masked language model accuracy. We make the model publicly available under the MIT open-source license.