# MM-HSD: Multi-Modal Hate Speech Detection in Videos

arXiv: https://arxiv.org/abs/2508.20546 · HTML: https://mlsanity.com/paper/2508.20546/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 72 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval](https://mlsanity.com/paper/2203.15086/) (2022)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [A Multimodal Framework for the Detection of Hateful Memes](https://mlsanity.com/paper/2012.12871/) (2020)
- [HateMM: A Multi-Modal Dataset for Hate Video Classification](https://mlsanity.com/paper/2305.03915/) (2023)
- [Detecting Hateful Memes Using a Multimodal Deep Ensemble](https://mlsanity.com/paper/2012.13235/) (2020)
- [Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons](https://mlsanity.com/paper/2103.10480/) (2021)
- [Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models](https://mlsanity.com/paper/2401.16727/) (2024)
- [MultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibili](https://mlsanity.com/paper/2408.03468/) (2024)
- [Temporal and cross-modal attention for audio-visual zero-shot learning](https://mlsanity.com/paper/2207.09966/) (2022)
- [Emotion Based Hate Speech Detection using Multimodal Learning](https://mlsanity.com/paper/2202.06218/) (2022)
- [A Web of Hate: Tackling Hateful Speech in Online Social Spaces](https://mlsanity.com/paper/1709.10159/) (2017)
- [Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.13659/) (2024)
- [ToxVidLM: A Multimodal Framework for Toxicity Detection in Code-Mixed Videos](https://mlsanity.com/paper/2405.20628/) (2024)
- [Cross-Modal Transfer from Memes to Videos: Addressing Data Scarcity in Hateful Video Detection](https://mlsanity.com/paper/2501.15438/) (2025)
- [Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content](https://mlsanity.com/paper/2502.07138/) (2025)
- [Multimodal Classification for Analysing Social Media](https://mlsanity.com/paper/1708.02099/) (2017)
- [Hateful Memes Detection via Complementary Visual and Linguistic Networks](https://mlsanity.com/paper/2012.04977/) (2020)
- [Unsupervised Cyberbullying Detection via Time-Informed Gaussian Mixture Model](https://mlsanity.com/paper/2008.02642/) (2020)
- [Two-Stream Video Classification with Cross-Modality Attention](https://mlsanity.com/paper/1908.00497/) (2019)
- [An Investigation Into Explainable Audio Hate Speech Detection](https://mlsanity.com/paper/2408.06065/) (2024)
- [Towards Patronizing and Condescending Language in Chinese Videos: A Multimodal Dataset and Detector](https://mlsanity.com/paper/2409.05005/) (2024)

## Built on by (most cited first, 5 of 5)

- [MultiHateLoc: Towards Temporal Localisation of Multimodal Hate Content in Online Videos](https://mlsanity.com/paper/2512.10408/) (2025)
- [Reasoning-Aware Multimodal Fusion for Hateful Video Detection](https://mlsanity.com/paper/2512.02743/) (2025)
- [Towards Training-free Multimodal Hate Localisation with Large Language Models](https://mlsanity.com/paper/2602.09637/) (2026)
- [Decoding Multimodal Cues: Unveiling the Implicit Meaning Behind Hateful Videos](https://mlsanity.com/paper/2606.11953/) (2026)
- [CLARA: Clip-Level Multimodal Alignment with VLM-Derived Rationales for Hateful Video Detection](https://mlsanity.com/paper/2608.15905/) (2026)

## Abstract (arXiv)

While hate speech detection (HSD) has been extensively studied in text, existing multi-modal approaches remain limited, particularly in videos. As modalities are not always individually informative, simple fusion methods fail to fully capture inter-modal dependencies. Moreover, previous work often omits relevant modalities such as on-screen text and audio, which may contain subtle hateful content and thus provide essential cues, both individually and in combination with others. In this paper, we present MM-HSD, a multi-modal model for HSD in videos that integrates video frames, audio, and text derived from speech transcripts and from frames (i.e.~on-screen text) together with features extracted by Cross-Modal Attention (CMA). We are the first to use CMA as an early feature extractor for HSD in videos, to systematically compare query/key configurations, and to evaluate the interactions between different modalities in the CMA block. Our approach leads to improved performance when on-screen text is used as a query and the rest of the modalities serve as a key. Experiments on the HateMM dataset show that MM-HSD outperforms state-of-the-art methods on M-F1 score (0.874), using concatenation of transcript, audio, video, on-screen text, and CMA for feature extraction on raw embeddings of the modalities. The code is available at https://github.com/idiap/mm-hsd