# B"uy"uk dil modellerinin T"urkcce verisetleri ile eugitilmesi ve ince ayarlanmasi

arXiv: https://arxiv.org/abs/2306.03978 · HTML: https://mlsanity.com/paper/2306.03978/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)

## Abstract (arXiv)

Large language models have advanced enormously, gained vast attraction and are having a phase of intensed research. Some of the developed models and training datasets have been made open-accessible. Hence these may be further fine-tuned with some techniques to obtain specialized models for specific tasks. When it comes to Turkish language, open-access models do not provide satisfactory coverage. This is also observed over published datasets. In this work, we propose some ideas to mitigate this issue: creating large Turkish datasets, training LLMs with these and fine-tuning pre-trained models with Turkish inputs. We report our findings on Turkish-based trainings with the problems encountered along the way. We conclude with outcomes of these experiments and propose ideas for further works. -- B"uy"uk dil modelleri inanilmaz "olcc"ude gelicsmekte, b"uy"uk ilgi toplayarak ve "uzerlerinde yougun aracstirmalarin yapildiugi bir d"onemdedirler. Gelicstirilen modeller ve eugitimde kullanilan verisetlerinden bazilari accik ericsimli olarak sunulmaktadir. B"oylece ince ayarlama teknikleri uygulayarak "ozellecsmics g"orevler iccin ccalicsabilir modeller elde edilmektedir. T"urkcce s"oz konusu olduugunda bu modellerinin kapsayiciliugi yeterli d"uzeyde deugildir. Bu durum, yayimlanan verisetlerinde de g"ozlemlenebilir. Bunu acsmanin yollari T"urkcce iccerikli b"uy"uk verisetlerinin olucsturulmasi, b"uy"uk dil modellerinin bunlarla eugitilmesi ve "onceden eugitilmics modellerin T"urkcce girdilerle ince ayarlanmalari olabilir. Bu ccalicsmada accik ericsimli dil modelleri ve verisetleri "uzerinde durulmakta ve T"urkcce temelli bazi deneyler, karcsilacsilan sorunlar ve sonucclar irdelenmektedir.