# Tamil-Llama: A New Tamil Language Model Based on Llama 2

arXiv: https://arxiv.org/abs/2311.05845 · HTML: https://mlsanity.com/paper/2311.05845/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://mlsanity.com/paper/2010.14571/) (2020)
- [AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages](https://mlsanity.com/paper/2005.00085/) (2020)

## Built on by (most cited first, 30 of 32)

- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data](https://mlsanity.com/paper/2406.15053/) (2024)
- [Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus](https://mlsanity.com/paper/2410.14815/) (2024)
- [How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?](https://mlsanity.com/paper/2406.11477/) (2024)
- [Egalitarian Language Representation in Language Models: It All Begins with Tokenizers](https://mlsanity.com/paper/2409.11501/) (2024)
- [An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference](https://mlsanity.com/paper/2402.10712/) (2024)
- [PersianLLaMA: Towards Building First Persian Large Language Model](https://mlsanity.com/paper/2312.15713/) (2023)
- [IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding](https://mlsanity.com/paper/2501.15747/) (2025)
- [Krutrim LLM: Multilingual Foundational Model for over a Billion People](https://mlsanity.com/paper/2502.09642/) (2025)
- [Efficient Continual Pre-training of LLMs for Low-resource Languages](https://mlsanity.com/paper/2412.10244/) (2024)
- [Adapting Chat Language Models Using Only Target Unlabeled Language Data](https://mlsanity.com/paper/2412.11704/) (2024)
- [Chitranuvad: Adapting Multi-Lingual LLMs for Multimodal Translation](https://mlsanity.com/paper/2502.20420/) (2025)
- [It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs](https://mlsanity.com/paper/2508.19089/) (2025)
- [Decoding the Diversity: A Review of the Indic AI Research Landscape](https://mlsanity.com/paper/2406.09559/) (2024)
- [Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance](https://mlsanity.com/paper/2406.11139/) (2024)
- [Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning](https://mlsanity.com/paper/2505.09738/) (2025)
- [[WIP] Jailbreak Paradox: The Achilles' Heel of LLMs](https://mlsanity.com/paper/2406.12702/) (2024)
- [Krutrim LLM: A Novel Tokenization Strategy for Multilingual Indic Languages with Petabyte-Scale Data Processing](https://mlsanity.com/paper/2407.12481/) (2024)
- [Exploring Pretraining via Active Forgetting for Improving Cross Lingual Transfer for Decoder Language Models](https://mlsanity.com/paper/2410.16168/) (2024)
- [Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer](https://mlsanity.com/paper/2505.10945/) (2025)
- [BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages](https://mlsanity.com/paper/2511.10338/) (2025)
- [Suvach -- Generated Hindi QA benchmark](https://mlsanity.com/paper/2404.19254/) (2024)
- [sPhinX: Sample Efficient Multilingual Instruction Fine-Tuning Through N-shot Guided Prompting](https://mlsanity.com/paper/2407.09879/) (2024)
- [UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settings](https://mlsanity.com/paper/2502.16961/) (2025)
- [Improving Multilingual Capabilities with Cultural and Local Knowledge in Large Language Models While Enhancing Native Performance](https://mlsanity.com/paper/2504.09753/) (2025)
- [Vocabulary Customization for Efficient Domain-Specific LLM Deployment](https://mlsanity.com/paper/2509.26124/) (2025)
- [Large Language Models for Math Education in Low-Resource Languages: A Study in Sinhala and Tamil](https://mlsanity.com/paper/2602.14517/) (2026)
- [LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation](https://mlsanity.com/paper/2603.22629/) (2026)
- [Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions](https://mlsanity.com/paper/2606.30790/) (2026)
- [Rethinking Indic AI from a Lens of Cultural Heritage Preservation](https://mlsanity.com/paper/2607.06544/) (2026)

## Abstract (arXiv)

Language modeling has witnessed remarkable advancements in recent years, with Large Language Models (LLMs) like ChatGPT setting unparalleled benchmarks in human-like text generation. However, a prevailing limitation is the underrepresentation of languages like Tamil in these cutting-edge models, leading to suboptimal performance in diverse linguistic contexts. This paper addresses this lacuna, enhancing the open-source LLaMA model with an addition of 16,000 Tamil tokens, aiming to achieve superior text generation and comprehension in the Tamil language. We strategically employ the LoRA methodology for efficient model training on a comprehensive Tamil corpus, ensuring computational feasibility and model robustness. Moreover, we introduce a Tamil-translated version of the Alpaca dataset and a subset of the OpenOrca dataset tailored for instruction fine-tuning. Our results showcase significant performance improvements in Tamil text generation, with potential implications for the broader landscape of LLMs in Indian languages. We further underscore our commitment to open research by making our models, datasets, and code publicly accessible, fostering further innovations in language modeling.