# From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation

arXiv: https://arxiv.org/abs/2404.09138 · HTML: https://mlsanity.com/paper/2404.09138/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 46 references that are in the map.
- First posted on arXiv 14 Apr 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [Are Multilingual Models Effective in Code-Switching?](https://mlsanity.com/paper/2103.13309/) (2021)
- [LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language](https://mlsanity.com/paper/2312.09993/) (2023)
- [Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set](https://mlsanity.com/paper/2312.12624/) (2023)
- [Language Resources for Dutch Large Language Modelling](https://mlsanity.com/paper/2312.12852/) (2023)
- [VinaLLaMA: LLaMA-based Vietnamese Foundation Model](https://mlsanity.com/paper/2312.11011/) (2023)
- [Automated multilingual detection of Pro-Kremlin propaganda in newspapers and Telegram posts](https://mlsanity.com/paper/2301.10604/) (2023)

## Built on by (most cited first, 6 of 6)

- [Decoding Hate: Exploring Language Models' Reactions to Hate Speech](https://mlsanity.com/paper/2410.00775/) (2024)
- [Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domains](https://mlsanity.com/paper/2411.14647/) (2024)
- [Empowering Smaller Models: Tuning LLaMA and Gemma with Chain-of-Thought for Ukrainian Exam Tasks](https://mlsanity.com/paper/2503.13988/) (2025)
- [Vuyko Mistral: Adapting LLMs for Low-Resource Dialectal Translation](https://mlsanity.com/paper/2506.07617/) (2025)
- [Hidden Persuasion: Detecting Manipulative Narratives on Social Media During the 2022 Russian Invasion of Ukraine](https://mlsanity.com/paper/2505.24028/) (2025)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

In the rapidly advancing field of AI and NLP, generative large language models (LLMs) stand at the forefront of innovation, showcasing unparalleled abilities in text understanding and generation. However, the limited representation of low-resource languages like Ukrainian poses a notable challenge, restricting the reach and relevance of this technology. Our paper addresses this by fine-tuning the open-source Gemma and Mistral LLMs with Ukrainian datasets, aiming to improve their linguistic proficiency and benchmarking them against other existing models capable of processing Ukrainian language. This endeavor not only aims to mitigate language bias in technology but also promotes inclusivity in the digital realm. Our transparent and reproducible approach encourages further NLP research and development. Additionally, we present the Ukrainian Knowledge and Instruction Dataset (UKID) to aid future efforts in language model fine-tuning. Our research not only advances the field of NLP but also highlights the importance of linguistic diversity in AI, which is crucial for cultural preservation, education, and expanding AI's global utility. Ultimately, we advocate for a future where technology is inclusive, enabling AI to communicate effectively across all languages, especially those currently underrepresented.