# Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection

arXiv: https://arxiv.org/abs/2510.16985 · HTML: https://mlsanity.com/paper/2510.16985/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 19 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla](https://mlsanity.com/paper/2101.00204/) (2021)
- [Classification Benchmarks for Under-resourced Bengali Language based on Multichannel Convolutional-LSTM Network](https://mlsanity.com/paper/2004.07807/) (2020)
- [DeepHateExplainer: Explainable Hate Speech Detection in Under-resourced Bengali Language](https://mlsanity.com/paper/2012.14353/) (2020)
- [Zero- and Few-Shot Prompting with LLMs: A Comparative Study with Fine-tuned Models for Bangla Sentiment Analysis](https://mlsanity.com/paper/2308.10783/) (2023)
- [Hate Speech and Offensive Language Detection in Bengali](https://mlsanity.com/paper/2210.03479/) (2022)
- [Multimodal Hate Speech Detection from Bengali Memes and Texts](https://mlsanity.com/paper/2204.10196/) (2022)
- [BD-SHS: A Benchmark Dataset for Learning to Detect Online Bangla Hate Speech in Different Social Contexts](https://mlsanity.com/paper/2206.00372/) (2022)
- [LLMem: Estimating GPU Memory Usage for Fine-Tuning Pre-Trained LLMs](https://mlsanity.com/paper/2404.10933/) (2024)
- [Model-Agnostic Meta-Learning for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2303.02513/) (2023)
- [Bangla Text Classification using Transformers](https://mlsanity.com/paper/2011.04446/) (2020)
- [HateTinyLLM : Hate Speech Detection Using Tiny Large Language Models](https://mlsanity.com/paper/2405.01577/) (2024)
- [LLMsAgainstHate @ NLU of Devanagari Script Languages 2025: Hate Speech Detection and Target Identification in Devanagari Languages via Parameter Efficient Fine-Tuning of LLMs](https://mlsanity.com/paper/2412.17131/) (2024)

## Built on by (most cited first, 3 of 3)

- [Quantifying Affective Bias in Low-Resource Media: Large-Scale Emotion Profiling of Bengali Headlines](https://mlsanity.com/paper/2510.17252/) (2025)
- [BengaliSent140: A Large-Scale Bengali Binary Sentiment Dataset for Hate and Non-Hate Speech Classification](https://mlsanity.com/paper/2601.20129/) (2026)
- [PermitGPT: A Unified Generative-AI Pipeline for Construction Hazard Forecasting, Permit Prediction, and Community Impact](https://mlsanity.com/paper/2608.28728/) (2026)

## Abstract (arXiv)

Bengali social media platforms have witnessed a sharp increase in hate speech, disproportionately affecting women and adolescents. While datasets such as BD-SHS provide a basis for structured evaluation, most prior approaches rely on either computationally costly full-model fine-tuning or proprietary APIs. This paper presents the first application of Parameter-Efficient Fine-Tuning (PEFT) for Bengali hate speech detection using LoRA and QLoRA. Three instruction-tuned large language models - Gemma-3-4B, Llama-3.2-3B, and Mistral-7B - were fine-tuned on the BD-SHS dataset of 50,281 annotated comments. Each model was adapted by training fewer than 1% of its parameters, enabling experiments on a single consumer-grade GPU. The results show that Llama-3.2-3B achieved the highest F1-score of 92.23%, followed by Mistral-7B at 88.94% and Gemma-3-4B at 80.25%. These findings establish PEFT as a practical and replicable strategy for Bengali and related low-resource languages.