# Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge

arXiv: https://arxiv.org/abs/2012.01002 · HTML: https://mlsanity.com/paper/2012.01002/

## Key facts

- Cited by 7 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 2 Dec 2020.

## Built on by (most cited first, 7 of 7)

- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [DISARM: Detecting the Victims Targeted by Harmful Memes](https://mlsanity.com/paper/2205.05738/) (2022)
- [Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset](https://mlsanity.com/paper/2107.04313/) (2021)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [What is Multimodality?](https://mlsanity.com/paper/2103.06304/) (2021)
- [RMS@CC-MMD 2026: Multimodal Misogyny Detection via Geometric Interaction and Multi-View Consensus](https://mlsanity.com/paper/2607.22709/) (2026)

## Abstract (arXiv)

Hateful Memes is a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. Difficult examples are added to the dataset to make it hard to rely on unimodal signals, which means only multimodal models can succeed. According to Kiela,the state-of-the-art methods perform poorly compared to humans (64.73% vs. 84.7% accuracy) on Hateful Memes. I propose a new model that combined multimodal with rules, which achieve the first ranking of accuracy and AUROC of 86.8% and 0.923 respectively. These rules are extracted from training set, and focus on improving the classification accuracy of difficult samples.