# HateTinyLLM : Hate Speech Detection Using Tiny Large Language Models

arXiv: https://arxiv.org/abs/2405.01577 · HTML: https://mlsanity.com/paper/2405.01577/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media](https://mlsanity.com/paper/1910.12574/) (2019)
- [An Investigation of Large Language Models for Real-World Hate Speech Detection](https://mlsanity.com/paper/2401.03346/) (2024)
- [Deep Learning for Hate Speech Detection: A Comparative Study](https://mlsanity.com/paper/2202.09517/) (2022)
- [Model-Agnostic Meta-Learning for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2303.02513/) (2023)
- [Generative AI for Hate Speech Detection: Evaluation and Findings](https://mlsanity.com/paper/2311.09993/) (2023)

## Built on by (most cited first, 7 of 7)

- [STAND-Guard: A Small Task-Adaptive Content Moderation Model](https://mlsanity.com/paper/2411.05214/) (2024)
- [Evaluation of Hate Speech Detection Using Large Language Models and Geographical Contextualization](https://mlsanity.com/paper/2502.19612/) (2025)
- [Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection](https://mlsanity.com/paper/2510.16985/) (2025)
- [Artificial Conversations, Real Results: Fostering Language Detection with Synthetic Data](https://mlsanity.com/paper/2503.24062/) (2025)
- [EdgeAIGuard: Agentic LLMs for Minor Protection in Digital Spaces](https://mlsanity.com/paper/2503.00092/) (2025)
- [Decoding Islamophobic Discourse: Using LLMs to Identify Tropes and Semi-Coded Hate Speech](https://mlsanity.com/paper/2503.18273/) (2025)
- [From Abusive Language Classification to Sequence Labeling Identification](https://mlsanity.com/paper/2610.06287/) (2026)

## Abstract (arXiv)

Hate speech encompasses verbal, written, or behavioral communication that targets derogatory or discriminatory language against individuals or groups based on sensitive characteristics. Automated hate speech detection plays a crucial role in curbing its propagation, especially across social media platforms. Various methods, including recent advancements in deep learning, have been devised to address this challenge. In this study, we introduce HateTinyLLM, a novel framework based on fine-tuned decoder-only tiny large language models (tinyLLMs) for efficient hate speech detection. Our experimental findings demonstrate that the fine-tuned HateTinyLLM outperforms the pretrained mixtral-7b model by a significant margin. We explored various tiny LLMs, including PY007/TinyLlama-1.1B-step-50K-105b, Microsoft/phi-2, and facebook/opt-1.3b, and fine-tuned them using LoRA and adapter methods. Our observations indicate that all LoRA-based fine-tuned models achieved over 80% accuracy.