# Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set

arXiv: https://arxiv.org/abs/2312.12624 · HTML: https://mlsanity.com/paper/2312.12624/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models](https://mlsanity.com/paper/2209.13325/) (2022)
- [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://mlsanity.com/paper/2104.05596/) (2021)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [Explaining Predictions of Non-Linear Classifiers in NLP](https://mlsanity.com/paper/1606.07298/) (2016)
- [BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text](https://mlsanity.com/paper/2109.14250/) (2021)
- [Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation](https://mlsanity.com/paper/2308.16797/) (2023)
- [How to decay your learning rate](https://mlsanity.com/paper/2103.12682/) (2021)
- [Universal Dependency Treebank for Odia Language](https://mlsanity.com/paper/2205.11976/) (2022)

## Built on by (most cited first, 12 of 12)

- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Krutrim LLM: Multilingual Foundational Model for over a Billion People](https://mlsanity.com/paper/2502.09642/) (2025)
- [Chitrarth: Bridging Vision and Language for a Billion People](https://mlsanity.com/paper/2502.15392/) (2025)
- [Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progress](https://mlsanity.com/paper/2408.14960/) (2024)
- [From Bytes to Borsch: Fine-Tuning Gemma and Mistral for the Ukrainian Language Representation](https://mlsanity.com/paper/2404.09138/) (2024)
- [LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language](https://mlsanity.com/paper/2405.07745/) (2024)
- [Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation](https://mlsanity.com/paper/2502.20420/) (2025)
- [From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages](https://mlsanity.com/paper/2410.18836/) (2024)
- [Decoding the Diversity: A Review of the Indic AI Research Landscape](https://mlsanity.com/paper/2406.09559/) (2024)
- [BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages](https://mlsanity.com/paper/2511.10338/) (2025)
- [Towards a More Inclusive AI: Progress and Perspectives in Large Language Model Training for the S'ami Language](https://mlsanity.com/paper/2405.05777/) (2024)
- [Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages](https://mlsanity.com/paper/2603.23521/) (2026)

## Abstract (arXiv)

Building LLMs for languages other than English is in great demand due to the unavailability and performance of multilingual LLMs, such as understanding the local context. The problem is critical for low-resource languages due to the need for instruction sets. In a multilingual country like India, there is a need for LLMs supporting Indic languages to provide generative AI and LLM-based technologies and services to its citizens. This paper presents our approach of i) generating a large Odia instruction set, including domain knowledge data suitable for LLM fine-tuning, and ii) building a Llama2-finetuned model tailored for enhanced performance in the Odia domain. The proposed work will help researchers build an instruction set and LLM, particularly for Indic languages. We will release the model and instruction set for the public for research and noncommercial purposes.