# A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models

arXiv: https://arxiv.org/abs/2107.03844 · HTML: https://mlsanity.com/paper/2107.03844/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 206 references that are in the map.
- First posted on arXiv 8 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018)
- [Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translation](https://mlsanity.com/paper/2009.09359/) (2020)
- [AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages](https://mlsanity.com/paper/2005.00085/) (2020)
- [Classification Benchmarks for Under-resourced Bengali Language based on Multichannel Convolutional-LSTM Network](https://mlsanity.com/paper/2004.07807/) (2020)
- [Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages](https://mlsanity.com/paper/2011.02323/) (2020)
- [Punctuation Prediction Model for Conversational Speech](https://mlsanity.com/paper/1807.00543/) (2018)
- [Bangla Text Classification using Transformers](https://mlsanity.com/paper/2011.04446/) (2020)
- [Sentiment Classification in Bangla Textual Content: A Comparative Study](https://mlsanity.com/paper/2011.10106/) (2020)
- [Sentiment Analysis on Bangla and Romanized Bangla Text (BRBT) using Deep Recurrent models](https://mlsanity.com/paper/1610.00369/) (2016)
- [Supervised learning Methods for Bangla Web Document Categorization](https://mlsanity.com/paper/1410.2045/) (2014)
- [Authorship Attribution in Bangla literature using Character-level CNN](https://mlsanity.com/paper/2001.05316/) (2020)
- [A Subword Level Language Model for Bangla Language](https://mlsanity.com/paper/1911.07613/) (2019)
- [Augmenting Statistical Machine Translation with Subword Translation of Out-of-Vocabulary Words](https://mlsanity.com/paper/1808.05700/) (2018)

## Built on by (most cited first, 13 of 13)

- [Zero- and Few-Shot Prompting with LLMs: A Comparative Study with Fine-tuned Models for Bangla Sentiment Analysis](https://mlsanity.com/paper/2308.10783/) (2023)
- [BLP-2023 Task 2: Sentiment Analysis](https://mlsanity.com/paper/2310.16183/) (2023)
- [Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings](https://mlsanity.com/paper/2408.02237/) (2024)
- [Intent Detection and Slot Filling for Home Assistants: Dataset and Analysis for Bangla and Sylheti](https://mlsanity.com/paper/2310.10935/) (2023)
- [Z-Index at CheckThat! Lab 2022: Check-Worthiness Identification on Tweet Text](https://mlsanity.com/paper/2207.07308/) (2022)
- [RSM-NLP at BLP-2023 Task 2: Bangla Sentiment Analysis using Weighted and Majority Voted Fine-Tuned Transformers](https://mlsanity.com/paper/2310.14261/) (2023)
- [Milestones in Bengali Sentiment Analysis leveraging Transformer-models: Fundamentals, Challenges and Future Directions](https://mlsanity.com/paper/2401.07847/) (2024)
- [A Systematic Survey of Natural Language Processing for the Greek Language](https://mlsanity.com/paper/2407.09861/) (2024)
- [Transformer-Based Low-Resource Language Translation: A Study on Standard Bengali to Sylheti](https://mlsanity.com/paper/2510.18898/) (2025)
- [Survey: Transformer-based Models in Data Modality Conversion](https://mlsanity.com/paper/2408.04723/) (2024)
- [Retrieval Augmented Enhanced Dual Co-Attention Framework for Target Aware Multimodal Bengali Hateful Meme Detection](https://mlsanity.com/paper/2602.19212/) (2026)
- [Multilingual Financial Fraud Detection Using Machine Learning and Transformer Models: A Bangla-English Study](https://mlsanity.com/paper/2603.11358/) (2026)
- [BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning](https://mlsanity.com/paper/2608.05104/) (2026)

## Abstract (arXiv)

Bangla -- ranked as the 6th most widely spoken language across the world (https://www.ethnologue.com/guides/ethnologue200), with 230 million native speakers -- is still considered as a low-resource language in the natural language processing (NLP) community. With three decades of research, Bangla NLP (BNLP) is still lagging behind mainly due to the scarcity of resources and the challenges that come with it. There is sparse work in different areas of BNLP; however, a thorough survey reporting previous work and recent advances is yet to be done. In this study, we first provide a review of Bangla NLP tasks, resources, and tools available to the research community; we benchmark datasets collected from various platforms for nine NLP tasks using current state-of-the-art algorithms (i.e., transformer-based models). We provide comparative results for the studied NLP tasks by comparing monolingual vs. multilingual models of varying sizes. We report our results using both individual and consolidated datasets and provide data splits for future research. We reviewed a total of 108 papers and conducted 175 sets of experiments. Our results show promising performance using transformer-based models while highlighting the trade-off with computational costs. We hope that such a comprehensive survey will motivate the community to build on and further advance the research on Bangla NLP.