# Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content

arXiv: https://arxiv.org/abs/2502.07138 · HTML: https://mlsanity.com/paper/2502.07138/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 46 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features](https://mlsanity.com/paper/2210.05916/) (2022)
- [HateMM: A Multi-Modal Dataset for Hate Video Classification](https://mlsanity.com/paper/2305.03915/) (2023)
- [Decoding the Underlying Meaning of Multimodal Hateful Memes](https://mlsanity.com/paper/2305.17678/) (2023)
- [MultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibili](https://mlsanity.com/paper/2408.03468/) (2024)
- [What is Gab? A Bastion of Free Speech or an Alt-Right Echo Chamber?](https://mlsanity.com/paper/1802.05287/) (2018)
- [Me, My Echo Chamber, and I: Introspection on Social Media Polarization](https://mlsanity.com/paper/1803.01731/) (2018)
- [Mapping Memes to Words for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2310.08368/) (2023)
- [Audio style transfer](https://mlsanity.com/paper/1710.11385/) (2017)
- [Modularized Networks for Few-shot Hateful Meme Detection](https://mlsanity.com/paper/2402.11845/) (2024)
- [Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection](https://mlsanity.com/paper/2407.21004/) (2024)

## Built on by (most cited first, 9 of 9)

- [MM-HSD: Multi-Modal Hate Speech Detection in Videos](https://mlsanity.com/paper/2508.20546/) (2025)
- [TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection](https://mlsanity.com/paper/2504.17902/) (2025)
- [MultiHateLoc: Towards Temporal Localisation of Multimodal Hate Content in Online Videos](https://mlsanity.com/paper/2512.10408/) (2025)
- [Shedding the Facades, Connecting the Domains: Detecting Shifting Multimodal Hate Video with Test-Time Adaptation](https://mlsanity.com/paper/2602.00132/) (2026)
- [Reasoning-Aware Multimodal Fusion for Hateful Video Detection](https://mlsanity.com/paper/2512.02743/) (2025)
- [TANDEM: Temporal-Aware Neural Detection for Multimodal Hate Speech](https://mlsanity.com/paper/2601.11178/) (2026)
- [Training-Free and Interpretable Hateful Video Detection via Multi-stage Adversarial Reasoning](https://mlsanity.com/paper/2601.15115/) (2026)
- [Decoding Multimodal Cues: Unveiling the Implicit Meaning Behind Hateful Videos](https://mlsanity.com/paper/2606.11953/) (2026)
- [Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection](https://mlsanity.com/paper/2609.18860/) (2026)

## Abstract (arXiv)

Social media platforms enable the propagation of hateful content across different modalities such as textual, auditory, and visual, necessitating effective detection methods. While recent approaches have shown promise in handling individual modalities, their effectiveness across different modality combinations remains unexplored. This paper presents a systematic analysis of fusion-based approaches for multimodal hate detection, focusing on their performance across video and image-based content. Our comprehensive evaluation reveals significant modality-specific limitations: while simple embedding fusion achieves state-of-the-art performance on video content (HateMM dataset) with a 9.9% points F1-score improvement, it struggles with complex image-text relationships in memes (Hateful Memes dataset). Through detailed ablation studies and error analysis, we demonstrate how current fusion approaches fail to capture nuanced cross-modal interactions, particularly in cases involving benign confounders. Our findings provide crucial insights for developing more robust hate detection systems and highlight the need for modality-specific architectural considerations. The code is available at https://github.com/gak97/Video-vs-Meme-Hate.