mlsanityData to 7 Oct 2026

Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus

arXiv 2410.14815 ↗First posted 18 Oct 2024cs.CL cs.LGcited by 24

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Multilingual LLMs support a variety of languages; however, their performance is suboptimal for low-resource languages. In this work, we emphasize the importance of continued pre-training of multilingual LLMs and the use of translation-based synthetic pre-training corpora for improving LLMs in low-resource languages. We conduct our study in the context of the low-resource Indic language Hindi. We introduce Nemotron-Mini-Hindi 4B, a bilingual SLM supporting both Hindi and English, based on Nemotron-Mini 4B. The model is trained using a mix of real and synthetic Hindi + English tokens, with continuous pre-training performed on 400B tokens. We demonstrate that both the base and instruct models achieve state-of-the-art results on Hindi benchmarks while remaining competitive on English tasks. Additionally, we observe that the continued pre-training approach enhances the model's overall factual accuracy. We perform an ablation study to highlight the impact of Hindi pre-training, showing significant improvements in Hindi chat capabilities and factual accuracy, which cannot be achieved through Hindi alignment alone.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 23 of its 27 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

23 of 27 references.

References · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  7. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  8. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
  9. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  10. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  11. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  12. 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 56
11 more
  1. 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
  2. 2023FinGPT: Large Generative Models for a Small Languagecited by 49
  3. 2022IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languagescited by 42
  4. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 40
  5. 2023Tamil-Llama: A New Tamil Language Model Based on Llama 2cited by 31
  6. 2022L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Librarycited by 30
  7. 2024INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languagescited by 24
  8. 2024Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapterscited by 14
  9. 2024L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Contextcited by 10
  10. 2024RedWhale: An Adapted Korean LLM Through Efficient Continual Pretrainingcited by 8
  11. 2024LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Languagecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 7.

24 papers.

Sort:

By area:

  1. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  2. 2025Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basquecited by 12
  3. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  4. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  5. 2024BgGPT 1.0: Extending English-centric LLMs to other languagescited by 7
  6. 2025PARAM-1 BharatGen 2.9B Modelcited by 5
  7. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  8. 2025Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Studycited by 4
  9. 2025Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysiscited by 2
  10. 2025Bridging Gaps in Hate Speech Detection: Meta-Collections and Benchmarks for Low-Resource Iberian Languagescited by 2
  11. 2025Improving Multilingual Capabilities with Cultural and Local Knowledge in Large Language Models While Enhancing Native Performancecited by 1
  12. 2025A Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge'ez Scriptcited by 1
12 more
  1. 2026EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Trainingcited by 1
  2. 2025Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languagescited by 0
  3. 2025FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Modelscited by 0
  4. 2025Teaching a Language Model to Speak the Language of Toolscited by 0
  5. 2025Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Modelscited by 0
  6. 2025Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lenscited by 0
  7. 2025Seeing Justice Clearly: Handwritten Legal Document Translation with OCR and Vision-Language Modelscited by 0
  8. 2026Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindicited by 0
  9. 2026Jupiter-N Technical Reportcited by 0
  10. 2026Source-Grounded Semantic Reinforcement Learning for Low-Resource Target-Language Generationcited by 0
  11. 2026Andha-Dhun: A First Look at Audio Descriptions in Hindicited by 0
  12. 2026Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extensioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs
  2. 2026Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
  3. 2025Scaling Low-Resource MT via Synthetic Data Generation with LLMs
  4. 2024Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages
  5. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  6. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
  7. 2024Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applications
  8. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models