# Benchmarking Parameter-Efficient Fine-Tuning of Large Language Models for Low-Resource Tajik Text Generation with the Tajik Web Corpus

arXiv: https://arxiv.org/abs/2605.03742 · HTML: https://mlsanity.com/paper/2605.03742/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 25 references that are in the map.
- First posted on arXiv 5 May 2026.

## Builds on (most cited first, 7 of 7)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [Cross-Lingual Transfer from Related Languages: Treating Low-Resource Maltese as Multilingual Code-Switching](https://mlsanity.com/paper/2401.16895/) (2024)
- [ParsTranslit: Truly Versatile Tajik-Farsi Transliteration](https://mlsanity.com/paper/2510.07520/) (2025)
- [Low-Resource, High-Impact: Building Corpora for Inclusive Language Technologies](https://mlsanity.com/paper/2512.14576/) (2025)
- [Tarab: A Multi-Dialect Corpus of Arabic Lyrics and Poetry](https://mlsanity.com/paper/2603.16601/) (2026)
- [Soro: A Lightweight Foundation Model and Chatbot for Tajik](https://mlsanity.com/paper/2605.27379/) (2026)

## Built on by (most cited first, 1 of 1)

- [Large Language Models for Low-Resource Languages: A Conceptual Framework for an Electronic Explanatory Dictionary of the Tajik Language](https://mlsanity.com/paper/2608.04186/) (2026)

## Abstract (arXiv)

We release the Tajik Web Corpus (319k docs, 1.11B chars) and benchmark generative LLMs on prompt continuation in Tajik, a low-resource Cyrillic-script language. Seventeen configurations across nine architectures are evaluated under three fine-tuning strategies: full fine-tuning, LoRA, and QLoRA (ranks 8 and 16). Because perplexity is not directly comparable across model families with different tokenizers, generation quality is assessed through perplexity interpreted within each family, complemented by qualitative analysis performed by a native Tajik speaker. Computational cost is measured via GPU memory and training time. The best quality-cost trade-off is achieved by Mistral 7B with QLoRA rank 8: perplexity 5.11 (within its tokenizer family), coherent Tajik output confirmed by the native speaker, 14.21 GB GPU memory, and approximately 33 minutes of training. Increasing the rank to 16 yields a negligible improvement for Mistral (perplexity 5.03, pairwise p > 0.05) while consuming about 1 GB more memory. Full fine-tuning of small GPT-2 models obtains lower numeric perplexity but leads to catastrophic forgetting (English or gibberish output); in contrast, QLoRA preserves multilingual pretrained knowledge and generates meaningful Tajik text. Encoder-only models perform worst (perplexity approximately 59), confirming their unsuitability for autoregressive generation. To our knowledge, this is the first systematic PEFT benchmark for Tajik text generation. Practical recommendations include using Mistral 7B with QLoRA r=8, avoiding full fine-tuning of small GPT-2 models, and adopting the released corpus and benchmark.