# Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

arXiv: https://arxiv.org/abs/2411.18571 · HTML: https://mlsanity.com/paper/2411.18571/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 12 of 12)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://mlsanity.com/paper/1909.04761/) (2019)
- [L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Library](https://mlsanity.com/paper/2205.14728/) (2022)
- [Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages](https://mlsanity.com/paper/2109.10534/) (2021)
- [Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?](https://mlsanity.com/paper/2404.14122/) (2024)
- [Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters](https://mlsanity.com/paper/2407.01406/) (2024)
- [adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds](https://mlsanity.com/paper/2403.02370/) (2024)
- [How Multilingual Are Large Language Models Fine-Tuned for Translation?](https://mlsanity.com/paper/2405.20512/) (2024)
- [Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies](https://mlsanity.com/paper/2404.09022/) (2024)

## Built on by (most cited first, 14 of 14)

- [Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges](https://mlsanity.com/paper/2509.11570/) (2025)
- [LingGym: How Far Are LLMs from Thinking Like Field Linguists?](https://mlsanity.com/paper/2511.00343/) (2025)
- [Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation](https://mlsanity.com/paper/2505.24174/) (2025)
- [Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safety](https://mlsanity.com/paper/2602.11157/) (2025)
- [Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir](https://mlsanity.com/paper/2605.04948/) (2026)
- [Multilingual LLMs Are Not Multilingual Thinkers: Evidence from Hindi Analogy Evaluation](https://mlsanity.com/paper/2507.13238/) (2025)
- [Improving Low-Resource Translation with Dictionary-Guided Fine-Tuning and RL: A Spanish-to-Wayuunaiki Study](https://mlsanity.com/paper/2508.19481/) (2025)
- [Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training](https://mlsanity.com/paper/2511.04485/) (2025)
- [Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructure](https://mlsanity.com/paper/2601.01244/) (2026)
- [Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Models](https://mlsanity.com/paper/2510.07203/) (2025)
- [UrduLM: A Resource-Efficient Monolingual Urdu Language Model](https://mlsanity.com/paper/2601.17664/) (2026)
- [The Hrunting of AI: Where and How to Improve English Dialectal Fairness](https://mlsanity.com/paper/2603.15187/) (2026)
- [Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuning](https://mlsanity.com/paper/2606.10428/) (2026)
- [Made in Hungary: Comments on the performance of generative language models](https://mlsanity.com/paper/2609.18284/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings.