BanglaByT5: Byte-Level Modelling for Bangla
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have achieved remarkable success across various natural language processing tasks. However, most LLM models use traditional tokenizers like BPE and SentencePiece, which fail to capture the finer nuances of a morphologically rich language like Bangla (Bengali). In this work, we introduce BanglaByT5, the first byte-level encoder-decoder model explicitly tailored for Bangla. Built upon a small variant of Googles ByT5 architecture, BanglaByT5 is pre-trained on a 14GB curated corpus combining high-quality literary and newspaper articles. Through zeroshot and supervised evaluations across generative and classification tasks, BanglaByT5 demonstrates competitive performance, surpassing several multilingual and larger models. Our findings highlight the efficacy of byte-level modelling for morphologically rich languages and highlight BanglaByT5 potential as a lightweight yet powerful tool for Bangla NLP, particularly in both resource-constrained and scalable environments.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 21 May 2025.
Builds on
29 of 39 references.
References · 29
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2018Deep contextualized word representationscited by 2,290
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
- 2020GLU Variants Improve Transformercited by 1,804
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
- 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
- 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
17 more
- 2016Exploring the Limits of Language Modelingcited by 173
- 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
- 2015Character-Aware Neural Language Modelscited by 132
- 2019Neural Machine Translation with Byte-Level Subwordscited by 115
- 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
- 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 80
- 2022BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Banglacited by 42
- 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
- 2016Character-based Neural Machine Translationcited by 33
- 2021Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitutioncited by 31
- 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
- 2016A Character-Level Decoder without Explicit Segmentation for Neural Machine Translationcited by 21
- 2021Training Multilingual Pre-trained Language Model with Byte-level Subwordscited by 16
- 2022BanglaParaphrase: A High-Quality Bangla Paraphrase Datasetcited by 15
- 2024One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Taskscited by 15
- 2023Vacaspati: A Diverse Corpus of Bangla Literaturecited by 10
- 2024Paramanu: Compact and Competitive Monolingual Language Models for Low-Resource Morphologically Rich Indian Languagescited by 5
Built on by
Citing papers per year: 2025: 2.
2 papers.