# UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settings

arXiv: https://arxiv.org/abs/2502.16961 · HTML: https://mlsanity.com/paper/2502.16961/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation](https://mlsanity.com/paper/2104.12369/) (2021)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [SeaLLMs -- Large Language Models for Southeast Asia](https://mlsanity.com/paper/2312.00738/) (2023)
- [TICO-19: the Translation Initiative for Covid-19](https://mlsanity.com/paper/2007.01788/) (2020)
- [Latxa: An Open Language Model and Evaluation Suite for Basque](https://mlsanity.com/paper/2403.20266/) (2024)
- [Tamil-Llama: A New Tamil Language Model Based on Llama 2](https://mlsanity.com/paper/2311.05845/) (2023)
- [Airavata: Introducing Hindi Instruction-tuned LLM](https://mlsanity.com/paper/2401.15006/) (2024)
- [TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities](https://mlsanity.com/paper/2212.06385/) (2022)
- [Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Results](https://mlsanity.com/paper/2303.04715/) (2023)
- [Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks](https://mlsanity.com/paper/2405.15453/) (2024)
- [Generalists vs. Specialists: Evaluating Large Language Models for Urdu](https://mlsanity.com/paper/2407.04459/) (2024)
- [Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-training](https://mlsanity.com/paper/2403.15470/) (2024)

## Built on by (most cited first, 1 of 1)

- [Multi-Task Learning by using Contextualized Word Representations for Syntactic Parsing of a Morphologically Rich Language](https://mlsanity.com/paper/2609.29855/) (2026)

## Abstract (arXiv)

Multilingual Large Language Models (LLMs) often provide suboptimal performance on low-resource languages like Urdu. This paper introduces UrduLLaMA 1.0, a model derived from the open-source Llama-3.1-8B-Instruct architecture and continually pre-trained on 128 million Urdu tokens, capturing the rich diversity of the language. To enhance instruction-following and translation capabilities, we leverage Low-Rank Adaptation (LoRA) to fine tune the model on 41,000 Urdu instructions and approximately 50,000 English-Urdu translation pairs. Evaluation across three machine translation datasets demonstrates significant performance improvements compared to state-of-the-art (SOTA) models, establishing a new benchmark for Urdu LLMs. These findings underscore the potential of targeted adaptation strategies with limited data and computational resources to address the unique challenges of low-resource languages.