# Exploring multi-task multi-lingual learning of transformer models for hate speech and offensive speech identification in social media

arXiv: https://arxiv.org/abs/2101.11155 · HTML: https://mlsanity.com/paper/2101.11155/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 67 references that are in the map.
- First posted on arXiv 27 Jan 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [Understanding Abuse: A Typology of Abusive Language Detection Subtasks](https://mlsanity.com/paper/1705.09899/) (2017)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media](https://mlsanity.com/paper/1910.12574/) (2019)
- [Multilingual Offensive Language Identification with Cross-lingual Embeddings](https://mlsanity.com/paper/2010.05324/) (2020)
- [QutNocturnal@HASOC'19: CNN for Hate Speech and Offensive Content Identification in Hindi Language](https://mlsanity.com/paper/2008.12448/) (2020)
- [Deep Multi-Task Learning with Shared Memory](https://mlsanity.com/paper/1609.07222/) (2016)
- [HateMonitors: Language Agnostic Abuse Detection in Social Media](https://mlsanity.com/paper/1909.12642/) (2019)

## Built on by (most cited first, 5 of 5)

- [A Survey of Machine Learning Models and Datasets for the Multi-label Classification of Textual Hate Speech in English](https://mlsanity.com/paper/2504.08609/) (2025)
- [Improved Multilingual Language Model Pretraining for Social Media Text via Translation Pair Prediction](https://mlsanity.com/paper/2110.10318/) (2021)
- [Multilingual Hate Speech Detection and Counterspeech Generation: A Comprehensive Survey and Practical Guide](https://mlsanity.com/paper/2603.19279/) (2026)
- [BOISHOMMO: Holistic Approach for Bangla Hate Speech](https://mlsanity.com/paper/2504.08408/) (2025)
- [Multi-task Learning with Active Learning for Arabic Offensive Speech Detection](https://mlsanity.com/paper/2506.02753/) (2025)

## Abstract (arXiv)

Hate Speech has become a major content moderation issue for online social media platforms. Given the volume and velocity of online content production, it is impossible to manually moderate hate speech related content on any platform. In this paper we utilize a multi-task and multi-lingual approach based on recently proposed Transformer Neural Networks to solve three sub-tasks for hate speech. These sub-tasks were part of the 2019 shared task on hate speech and offensive content (HASOC) identification in Indo-European languages. We expand on our submission to that competition by utilizing multi-task models which are trained using three approaches, a) multi-task learning with separate task heads, b) back-translation, and c) multi-lingual training. Finally, we investigate the performance of various models and identify instances where the Transformer based models perform differently and better. We show that it is possible to to utilize different combined approaches to obtain models that can generalize easily on different languages and tasks, while trading off slight accuracy (in some cases) for a much reduced inference time compute cost. We open source an updated version of our HASOC 2019 code with the new improvements at https://github.com/socialmediaie/MTML_HateSpeech.