# Exploring Hate Speech Detection in Multimodal Publications

arXiv: https://arxiv.org/abs/1910.03814 · HTML: https://mlsanity.com/paper/1910.03814/

## Key facts

- Cited by 103 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 30 of 103)

- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?](https://mlsanity.com/paper/2202.06675/) (2022)
- [Prompting for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2302.04156/) (2023)
- [Detecting and Understanding Harmful Memes: A Survey](https://mlsanity.com/paper/2205.04274/) (2022)
- [A Survey on Multimodal Disinformation Detection](https://mlsanity.com/paper/2103.12541/) (2021)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [HateMM: A Multi-Modal Dataset for Hate Video Classification](https://mlsanity.com/paper/2305.03915/) (2023)
- [On Explaining Multimodal Hateful Meme Detection Models](https://mlsanity.com/paper/2204.01734/) (2022)
- [The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress](https://mlsanity.com/paper/2104.07123/) (2021)
- [A systematic review of Hate Speech automatic detection using Natural Language Processing](https://mlsanity.com/paper/2106.00742/) (2021)
- [Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models](https://mlsanity.com/paper/2401.16727/) (2024)
- [The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements](https://mlsanity.com/paper/2101.06053/) (2021)
- ["Subverting the Jewtocracy": Online Antisemitism Detection Using Multimodal Deep Learning](https://mlsanity.com/paper/2104.05947/) (2021)
- [DISARM: Detecting the Victims Targeted by Harmful Memes](https://mlsanity.com/paper/2205.05738/) (2022)
- [BanglaAbuseMeme: A Dataset for Bengali Abusive Meme Classification](https://mlsanity.com/paper/2310.11748/) (2023)
- [LAraBench: Benchmarking Arabic AI with Large Language Models](https://mlsanity.com/paper/2305.14982/) (2023)
- [Multimodal Hate Speech Detection from Bengali Memes and Texts](https://mlsanity.com/paper/2204.10196/) (2022)
- [Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?](https://mlsanity.com/paper/2301.11219/) (2023)
- [Perturbations in the Wild: Leveraging Human-Written Text Perturbations for Realistic Adversarial Attack and Defense](https://mlsanity.com/paper/2203.10346/) (2022)
- [Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations](https://mlsanity.com/paper/2401.09899/) (2024)
- [Towards Comprehensive Detection of Chinese Harmful Memes](https://mlsanity.com/paper/2410.02378/) (2024)
- [Exploring Visual Engagement Signals for Representation Learning](https://mlsanity.com/paper/2104.07767/) (2021)
- [Memotion 3: Dataset on Sentiment and Emotion Analysis of Codemixed Hindi-English Memes](https://mlsanity.com/paper/2303.09892/) (2023)
- [SoMeLVLM: A Large Vision Language Model for Social Media Processing](https://mlsanity.com/paper/2402.13022/) (2024)
- [Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretations](https://mlsanity.com/paper/2402.12198/) (2024)
- [ToxVidLM: A Multimodal Framework for Toxicity Detection in Code-Mixed Videos](https://mlsanity.com/paper/2405.20628/) (2024)
- [BanTH: A Multi-label Hate Speech Detection Dataset for Transliterated Bangla](https://mlsanity.com/paper/2410.13281/) (2024)
- [Multimodal Fusion Refiner Networks](https://mlsanity.com/paper/2104.03435/) (2021)
- [Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modality](https://mlsanity.com/paper/2305.10547/) (2023)
- [Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Content](https://mlsanity.com/paper/2502.07138/) (2025)

## Abstract (arXiv)

In this work we target the problem of hate speech detection in multimodal publications formed by a text and an image. We gather and annotate a large scale dataset from Twitter, MMHS150K, and propose different models that jointly analyze textual and visual information for hate speech detection, comparing them with unimodal detection. We provide quantitative and qualitative results and analyze the challenges of the proposed task. We find that, even though images are useful for the hate speech detection task, current multimodal models cannot outperform models analyzing only text. We discuss why and open the field and the dataset for further research.