# Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus

arXiv: https://arxiv.org/abs/2410.14815 · HTML: https://mlsanity.com/paper/2410.14815/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 23 of its 27 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts](https://mlsanity.com/paper/2404.05993/) (2024)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning](https://mlsanity.com/paper/2404.00213/) (2024)
- [IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages](https://mlsanity.com/paper/2403.06350/) (2024)
- [FinGPT: Large Generative Models for a Small Language](https://mlsanity.com/paper/2311.05640/) (2023)
- [IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages](https://mlsanity.com/paper/2203.05437/) (2022)
- [NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment](https://mlsanity.com/paper/2405.01481/) (2024)
- [Tamil-Llama: A New Tamil Language Model Based on Llama 2](https://mlsanity.com/paper/2311.05845/) (2023)
- [L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Library](https://mlsanity.com/paper/2205.14728/) (2022)
- [INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages](https://mlsanity.com/paper/2407.13522/) (2024)
- [Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters](https://mlsanity.com/paper/2407.01406/) (2024)
- [L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Context](https://mlsanity.com/paper/2409.08706/) (2024)
- [RedWhale: An Adapted Korean LLM Through Efficient Continual Pretraining](https://mlsanity.com/paper/2408.11294/) (2024)
- [LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language](https://mlsanity.com/paper/2405.07745/) (2024)

## Built on by (most cited first, 24 of 24)

- [Knowledge Boundary of Large Language Models: A Survey](https://mlsanity.com/paper/2412.12472/) (2024)
- [Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque](https://mlsanity.com/paper/2506.07597/) (2025)
- [AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages](https://mlsanity.com/paper/2601.06395/) (2026)
- [Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi](https://mlsanity.com/paper/2504.06011/) (2025)
- [BgGPT 1.0: Extending English-centric LLMs to other languages](https://mlsanity.com/paper/2412.10893/) (2024)
- [PARAM-1 BharatGen 2.9B Model](https://mlsanity.com/paper/2507.13390/) (2025)
- [Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs](https://mlsanity.com/paper/2412.14471/) (2024)
- [Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study](https://mlsanity.com/paper/2507.14304/) (2025)
- [Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis](https://mlsanity.com/paper/2508.19831/) (2025)
- [Bridging Gaps in Hate Speech Detection: Meta-Collections and Benchmarks for Low-Resource Iberian Languages](https://mlsanity.com/paper/2510.11167/) (2025)
- [Improving Multilingual Capabilities with Cultural and Local Knowledge in Large Language Models While Enhancing Native Performance](https://mlsanity.com/paper/2504.09753/) (2025)
- [A Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge'ez Script](https://mlsanity.com/paper/2507.15142/) (2025)
- [EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training](https://mlsanity.com/paper/2603.02041/) (2026)
- [Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages](https://mlsanity.com/paper/2506.19187/) (2025)
- [FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models](https://mlsanity.com/paper/2506.21563/) (2025)
- [Teaching a Language Model to Speak the Language of Tools](https://mlsanity.com/paper/2506.23394/) (2025)
- [Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Models](https://mlsanity.com/paper/2510.07203/) (2025)
- [Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lens](https://mlsanity.com/paper/2511.03880/) (2025)
- [Seeing Justice Clearly: Handwritten Legal Document Translation with OCR and Vision-Language Models](https://mlsanity.com/paper/2512.18004/) (2025)
- [Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi](https://mlsanity.com/paper/2603.03508/) (2026)
- [Jupiter-N Technical Report](https://mlsanity.com/paper/2604.17429/) (2026)
- [Source-Grounded Semantic Reinforcement Learning for Low-Resource Target-Language Generation](https://mlsanity.com/paper/2605.29502/) (2026)
- [Andha-Dhun: A First Look at Audio Descriptions in Hindi](https://mlsanity.com/paper/2607.06457/) (2026)
- [Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension](https://mlsanity.com/paper/2608.03494/) (2026)

## Abstract (arXiv)

Multilingual LLMs support a variety of languages; however, their performance is suboptimal for low-resource languages. In this work, we emphasize the importance of continued pre-training of multilingual LLMs and the use of translation-based synthetic pre-training corpora for improving LLMs in low-resource languages. We conduct our study in the context of the low-resource Indic language Hindi. We introduce Nemotron-Mini-Hindi 4B, a bilingual SLM supporting both Hindi and English, based on Nemotron-Mini 4B. The model is trained using a mix of real and synthetic Hindi + English tokens, with continuous pre-training performed on 400B tokens. We demonstrate that both the base and instruct models achieve state-of-the-art results on Hindi benchmarks while remaining competitive on English tasks. Additionally, we observe that the continued pre-training approach enhances the model's overall factual accuracy. We perform an ablation study to highlight the impact of Hindi pre-training, showing significant improvements in Hindi chat capabilities and factual accuracy, which cannot be achieved through Hindi alignment alone.