# Fine-Tuning Large Language Models with QLoRA for Offensive Language Detection in Roman Urdu-English Code-Mixed Text

arXiv: https://arxiv.org/abs/2510.03683 · HTML: https://mlsanity.com/paper/2510.03683/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 4 Oct 2025.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion](https://mlsanity.com/paper/2203.13224/) (2022)
- [Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model](https://mlsanity.com/paper/2008.06460/) (2020)
- [Overview of the HASOC track at FIRE 2020: Hate Speech and Offensive Content Identification in Indo-European Languages](https://mlsanity.com/paper/2108.05927/) (2021)
- [HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning](https://mlsanity.com/paper/2311.00321/) (2023)
- [Don't Go To Extremes: Revealing the Excessive Sensitivity and Calibration Limitations of LLMs in Implicit Hate Speech Detection](https://mlsanity.com/paper/2402.11406/) (2024)
- [On Zero-Shot Counterspeech Generation by LLMs](https://mlsanity.com/paper/2403.14938/) (2024)
- [Outcome-Constrained Large Language Models for Countering Hate Speech](https://mlsanity.com/paper/2403.17146/) (2024)
- [Social-LLM: Modeling User Behavior at Scale using Language Models and Social Network Data](https://mlsanity.com/paper/2401.00893/) (2023)
- [Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations](https://mlsanity.com/paper/2403.06009/) (2024)
- [HateRephrase: Zero- and Few-Shot Reduction of Hate Intensity in Online Posts using Large Language Models](https://mlsanity.com/paper/2310.13985/) (2023)
- [Target Span Detection for Implicit Harmful Content](https://mlsanity.com/paper/2403.19836/) (2024)
- [LLM generated responses to mitigate the impact of hate speech](https://mlsanity.com/paper/2311.16905/) (2023)
- [Evaluating ChatGPT's Performance for Multilingual and Emoji-based Hate Speech Detection](https://mlsanity.com/paper/2305.13276/) (2023)
- [Using LLMs to discover emerging coded antisemitic hate-speech in extremist social media](https://mlsanity.com/paper/2401.10841/) (2024)

## Built on by (most cited first, 1 of 1)

- [Irony Detection in Urdu Text: A Comparative Study Using Machine Learning Models and Large Language Models](https://mlsanity.com/paper/2510.22356/) (2025)

## Abstract (arXiv)

The use of derogatory terms in languages that employ code mixing, such as Roman Urdu, presents challenges for Natural Language Processing systems due to unstated grammar, inconsistent spelling, and a scarcity of labeled data. In this work, we propose a QLoRA based fine tuning framework to improve offensive language detection in Roman Urdu-English text. We translated the Roman Urdu-English code mixed dataset into English using Google Translate to leverage English LLMs, while acknowledging that this translation reduces direct engagement with code mixing features. Our focus is on classification performance using English translated low resource inputs. We fine tuned several transformers and large language models, including Meta LLaMA 3 8B, Mistral 7B v0.1, LLaMA 2 7B, ModernBERT, and RoBERTa, with QLoRA for memory efficient adaptation. Models were trained and evaluated on a manually annotated Roman Urdu dataset for offensive vs non offensive content. Of all tested models, the highest F1 score of 91.45 was attained by Meta LLaMA 3 8B, followed by Mistral 7B at 89.66, surpassing traditional transformer baselines. These results demonstrate the efficacy of QLoRA in fine tuning high performing models for low resource environments such as code mixed offensive language detection, and confirm the potential of LLMs for this task. This work advances a scalable approach to Roman Urdu moderation and paves the way for future multilingual offensive detection systems based on LLMs.