# ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

arXiv: https://arxiv.org/abs/2405.14141 · HTML: https://mlsanity.com/paper/2405.14141/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media](https://mlsanity.com/paper/1910.12574/) (2019)
- [ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation](https://mlsanity.com/paper/2205.06457/) (2022)
- [FBERT: A Neural Transformer for Identifying Offensive Content](https://mlsanity.com/paper/2109.05074/) (2021)
- [A Large-scale Dataset for Hate Speech Detection on Vietnamese Social Media Texts](https://mlsanity.com/paper/2103.11528/) (2021)
- [Improving Sequence Tagging for Vietnamese Text Using Transformer-based Neural Models](https://mlsanity.com/paper/2006.15994/) (2020)
- [Constructive and Toxic Speech Detection for Open-domain Social Media Comments in Vietnamese](https://mlsanity.com/paper/2103.10069/) (2021)
- [ViHOS: Hate Speech Spans Detection for Vietnamese](https://mlsanity.com/paper/2301.10186/) (2023)
- [HSD Shared Task in VLSP Campaign 2019:Hate Speech Detection for Social Good](https://mlsanity.com/paper/2007.06493/) (2020)
- [ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing](https://mlsanity.com/paper/2310.11166/) (2023)
- [SMTCE: A Social Media Text Classification Evaluation Benchmark and BERTology Models for Vietnamese](https://mlsanity.com/paper/2209.10482/) (2022)
- [Tackling Hate Speech in Low-resource Languages with Context Experts](https://mlsanity.com/paper/2303.16828/) (2023)
- [A Text-to-Text Model for Multilingual Offensive Language Identification](https://mlsanity.com/paper/2312.03379/) (2023)

## Built on by (most cited first, 3 of 3)

- [On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMs](https://mlsanity.com/paper/2410.12600/) (2024)
- [ViToSA: Audio-Based Toxic Spans Detection on Vietnamese Speech Utterances](https://mlsanity.com/paper/2506.00636/) (2025)
- [NEU-ESC: A Comprehensive Vietnamese dataset for Educational Sentiment analysis and topic Classification toward multitask learning](https://mlsanity.com/paper/2506.23524/) (2025)

## Abstract (arXiv)

Recent advancements in hate speech detection (HSD) in Vietnamese have made significant progress, primarily attributed to the emergence of transformer-based pre-trained language models, particularly those built on the BERT architecture. However, the necessity for specialized fine-tuned models has resulted in the complexity and fragmentation of developing a multitasking HSD system. Moreover, most current methodologies focus on fine-tuning general pre-trained models, primarily trained on formal textual datasets like Wikipedia, which may not accurately capture human behavior on online platforms. In this research, we introduce ViHateT5, a T5-based model pre-trained on our proposed large-scale domain-specific dataset named VOZ-HSD. By harnessing the power of a text-to-text architecture, ViHateT5 can tackle multiple tasks using a unified model and achieve state-of-the-art performance across all standard HSD benchmarks in Vietnamese. Our experiments also underscore the significance of label distribution in pre-training data on model efficacy. We provide our experimental materials for research purposes, including the VOZ-HSD dataset, pre-trained checkpoint, the unified HSD-multitask ViHateT5 model, and related source code on GitHub publicly.