# BanglaLlama: LLaMA for Bangla Language

arXiv: https://arxiv.org/abs/2410.21200 · HTML: https://mlsanity.com/paper/2410.21200/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)

## Built on by (most cited first, 5 of 5)

- [Breaking the Silence: A Dataset and Benchmark for Bangla Text-to-Gloss Translation](https://mlsanity.com/paper/2504.02293/) (2025)
- [BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali](https://mlsanity.com/paper/2605.31483/) (2026)
- [Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms](https://mlsanity.com/paper/2608.22335/) (2026)
- [Reveal-Bangla: A Dataset for Cross-Lingual Multi-Step Reasoning Evaluation](https://mlsanity.com/paper/2508.08933/) (2025)
- [TRACE-BN: Transferring Bangla-English Tutoring Behavior to a Sub-1B Offline Language Model](https://mlsanity.com/paper/2608.15223/) (2026)

## Abstract (arXiv)

Bangla is a language spoken by approximately 240 million native speakers and around 300 million people worldwide. Despite being the 5th largest spoken language in the world, Bangla is still a "low-resource" language, and existing pretrained language models often struggle to perform well on Bangla Language Processing (BLP) tasks. This paper addresses this gap by: (1) introducing two high-quality translated Bangla-instruction datasets totaling 224k samples - Bangla-Orca (172k) and Bangla-Alpaca (52k); and (2) leveraging these datasets to develop BanglaLlama, an open-source family of Bangla-specific LLMs, consisting of five base and instruct variants. We present our methodology, two large datasets, and comprehensive benchmarking results showcasing the effectiveness of our dataset and model on multiple benchmarks. We believe our proposed datasets and models will serve as the new standard baseline for future research focused on this widely spoken yet "low-resource" language.