UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settings
Not on the map. Why?
Abstract (arXiv)
Multilingual Large Language Models (LLMs) often provide suboptimal performance on low-resource languages like Urdu. This paper introduces UrduLLaMA 1.0, a model derived from the open-source Llama-3.1-8B-Instruct architecture and continually pre-trained on 128 million Urdu tokens, capturing the rich diversity of the language. To enhance instruction-following and translation capabilities, we leverage Low-Rank Adaptation (LoRA) to fine tune the model on 41,000 Urdu instructions and approximately 50,000 English-Urdu translation pairs. Evaluation across three machine translation datasets demonstrates significant performance improvements compared to state-of-the-art (SOTA) models, establishing a new benchmark for Urdu LLMs. These findings underscore the potential of targeted adaptation strategies with limited data and computational resources to address the unique challenges of low-resource languages.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 24 Feb 2025.
Builds on
28 of 48 references.
References · 28
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
- 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
- 2021Deduplicating Training Data Makes Language Models Bettercited by 714
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
- 2024TinyLlama: An Open-Source Small Language Modelcited by 618
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
- 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
- 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
16 more
- 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
- 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
- 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
- 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
- 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
- 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 128
- 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
- 2020TICO-19: the Translation Initiative for Covid-19cited by 68
- 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
- 2023Tamil-Llama: A New Tamil Language Model Based on Llama 2cited by 32
- 2024Airavata: Introducing Hindi Instruction-tuned LLMcited by 32
- 2022TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalitiescited by 19
- 2023Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Resultscited by 10
- 2024Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Taskscited by 8
- 2024Generalists vs. Specialists: Evaluating Large Language Models for Urducited by 5
- 2024Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-trainingcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.