# Emotion Based Hate Speech Detection using Multimodal Learning

arXiv: https://arxiv.org/abs/2202.06218 · HTML: https://mlsanity.com/paper/2202.06218/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 65 references that are in the map.
- First posted on arXiv 13 Feb 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)](https://mlsanity.com/paper/2006.07235/) (2020)
- [Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes](https://mlsanity.com/paper/2012.07788/) (2020)
- [Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge](https://mlsanity.com/paper/2012.12975/) (2020)
- [Understanding the Behaviors of BERT in Ranking](https://mlsanity.com/paper/1904.07531/) (2019)
- [One-step and Two-step Classification for Abusive Language Detection on Twitter](https://mlsanity.com/paper/1706.01206/) (2017)
- [Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution](https://mlsanity.com/paper/2012.08290/) (2020)
- [Deep Multimodal Learning for Audio-Visual Speech Recognition](https://mlsanity.com/paper/1501.05396/) (2015)
- [Tackling Online Abuse: A Survey of Automated Abuse Detection Methods](https://mlsanity.com/paper/1908.06024/) (2019)
- [Attention Based Fully Convolutional Network for Speech Emotion Recognition](https://mlsanity.com/paper/1806.01506/) (2018)
- [Abusive Language Detection with Graph Convolutional Networks](https://mlsanity.com/paper/1904.04073/) (2019)
- [Deep Learning for User Comment Moderation](https://mlsanity.com/paper/1705.09993/) (2017)
- [UHH-LT at SemEval-2020 Task 12: Fine-Tuning of Pre-Trained Transformer Networks for Offensive Language Detection](https://mlsanity.com/paper/2004.11493/) (2020)

## Built on by (most cited first, 18 of 18)

- [HateMM: A Multi-Modal Dataset for Hate Video Classification](https://mlsanity.com/paper/2305.03915/) (2023)
- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [ToxVidLM: A Multimodal Framework for Toxicity Detection in Code-Mixed Videos](https://mlsanity.com/paper/2405.20628/) (2024)
- [GPT-4V(ision) as A Social Media Analysis Engine](https://mlsanity.com/paper/2311.07547/) (2023)
- [Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights](https://mlsanity.com/paper/2406.17430/) (2024)
- [MM-HSD: Multi-Modal Hate Speech Detection in Videos](https://mlsanity.com/paper/2508.20546/) (2025)
- [An Investigation Into Explainable Audio Hate Speech Detection](https://mlsanity.com/paper/2408.06065/) (2024)
- [ARC-NLP at Multimodal Hate Speech Event Detection 2023: Multimodal Methods Boosted by Ensemble Learning, Syntactical and Entity Features](https://mlsanity.com/paper/2307.13829/) (2023)
- [Attentive Fusion: A Transformer-based Approach to Multimodal Hate Speech Detection](https://mlsanity.com/paper/2401.10653/) (2024)
- [MHS-STMA: Multimodal Hate Speech Detection via Scalable Transformer-Based Multilevel Attention Framework](https://mlsanity.com/paper/2409.05136/) (2024)
- [Multilingual and Multimodal Abuse Detection](https://mlsanity.com/paper/2204.02263/) (2022)
- [Safe Guard: an LLM-agent for Real-time Voice-based Hate Speech Detection in Social Virtual Reality](https://mlsanity.com/paper/2409.15623/) (2024)
- [Towards Cross-Lingual Audio Abuse Detection in Low-Resource Settings with Few-Shot Learning](https://mlsanity.com/paper/2412.01408/) (2024)
- [Co-AttenDWG: Co-Attentive Dimension-Wise Gating and Expert Fusion for Multi-Modal Offensive Content Detection](https://mlsanity.com/paper/2505.19010/) (2025)
- [Multi-task Learning with Active Learning for Arabic Offensive Speech Detection](https://mlsanity.com/paper/2506.02753/) (2025)
- [Bangla Hate Speech Classification with Fine-tuned Transformer Models](https://mlsanity.com/paper/2512.02845/) (2025)
- [Beyond Content: A Comprehensive Speech Toxicity Dataset and Detection Framework Incorporating Paralinguistic Cues](https://mlsanity.com/paper/2605.15984/) (2026)
- [MexHat: A Dataset for Hate Speech Detection in Mexican Spanish Videos](https://mlsanity.com/paper/2609.31553/) (2026)

## Abstract (arXiv)

In recent years, monitoring hate speech and offensive language on social media platforms has become paramount due to its widespread usage among all age groups, races, and ethnicities. Consequently, there have been substantial research efforts towards automated detection of such content using Natural Language Processing (NLP). While successfully filtering textual data, no research has focused on detecting hateful content in multimedia data. With increased ease of data storage and the exponential growth of social media platforms, multimedia content proliferates the internet as much as text data. Nevertheless, it escapes the automatic filtering systems. Hate speech and offensiveness can be detected in multimedia primarily via three modalities, i.e., visual, acoustic, and verbal. Our preliminary study concluded that the most essential features in classifying hate speech would be the speaker's emotional state and its influence on the spoken words, therefore limiting our current research to these modalities. This paper proposes the first multimodal deep learning framework to combine the auditory features representing emotion and the semantic features to detect hateful content. Our results demonstrate that incorporating emotional attributes leads to significant improvement over text-based models in detecting hateful multimedia content. This paper also presents a new Hate Speech Detection Video Dataset (HSDVD) collected for the purpose of multimodal learning as no such dataset exists today.