mlsanityData to 14 Oct 2026

Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

arXiv 2411.18571 ↗First posted 27 Nov 2024cs.CL cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 12 of its 19 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

12 of 19 references.

References · 12

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  3. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  4. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  5. 2019MultiFiT: Efficient Multi-lingual Language Model Fine-tuningcited by 31
  6. 2022L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Librarycited by 31
  7. 2021Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languagescited by 24
  8. 2024Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?cited by 17
  9. 2024Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapterscited by 15
  10. 2024adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgroundscited by 14
  11. 2024How Multilingual Are Large Language Models Fine-Tuned for Translation?cited by 6
  12. 2024Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategiescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challengescited by 6
  2. 2025LingGym: How Far Are LLMs from Thinking Like Field Linguists?cited by 6
  3. 2025Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generationcited by 5
  4. 2025Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safetycited by 4
  5. 2026Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkircited by 3
  6. 2025Multilingual LLMs Are Not Multilingual Thinkers: Evidence from Hindi Analogy Evaluationcited by 1
  7. 2025Improving Low-Resource Translation with Dictionary-Guided Fine-Tuning and RL: A Spanish-to-Wayuunaiki Studycited by 1
  8. 2025Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Trainingcited by 1
  9. 2026Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructurecited by 1
  10. 2025Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Modelscited by 0
  11. 2026UrduLM: A Resource-Efficient Monolingual Urdu Language Modelcited by 0
  12. 2026The Hrunting of AI: Where and How to Improve English Dialectal Fairnesscited by 0
2 more
  1. 2026Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuningcited by 0
  2. 2026Made in Hungary: Comments on the performance of generative language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LoRA Is Slower Than You Think
  2. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
  3. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
  4. 2025Dual LoRA: Enhancing LoRA with Magnitude and Direction Updates
  5. 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
  6. 2025Slimming Down LLMs Without Losing Their Minds
  7. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  8. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation