# Detecting Hate Speech in Multi-modal Memes

arXiv: https://arxiv.org/abs/2012.14891 · HTML: https://mlsanity.com/paper/2012.14891/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2020.

## Built on by (most cited first, 30 of 38)

- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [A Survey on Multimodal Disinformation Detection](https://mlsanity.com/paper/2103.12541/) (2021)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [Detecting Propaganda Techniques in Memes](https://mlsanity.com/paper/2109.08013/) (2021)
- [HateMM: A Multi-Modal Dataset for Hate Video Classification](https://mlsanity.com/paper/2305.03915/) (2023)
- [Decoding the Underlying Meaning of Multimodal Hateful Memes](https://mlsanity.com/paper/2305.17678/) (2023)
- [Towards Low-Resource Harmful Meme Detection with LMM Agents](https://mlsanity.com/paper/2411.05383/) (2024)
- [IRFL: Image Recognition of Figurative Language](https://mlsanity.com/paper/2303.15445/) (2023)
- [DISARM: Detecting the Victims Targeted by Harmful Memes](https://mlsanity.com/paper/2205.05738/) (2022)
- [Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models](https://mlsanity.com/paper/2401.13298/) (2024)
- [Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models](https://mlsanity.com/paper/2312.05434/) (2023)
- [Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?](https://mlsanity.com/paper/2301.11219/) (2023)
- [Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset](https://mlsanity.com/paper/2107.04313/) (2021)
- [How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?](https://mlsanity.com/paper/2404.12866/) (2024)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [SoMeLVLM: A Large Vision Language Model for Social Media Processing](https://mlsanity.com/paper/2402.13022/) (2024)
- [Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modality](https://mlsanity.com/paper/2305.10547/) (2023)
- [AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness](https://mlsanity.com/paper/2507.01702/) (2025)
- [MemeGraphs: Linking Memes to Knowledge Graphs](https://mlsanity.com/paper/2305.18391/) (2023)
- [GPT-4V(ision) as A Social Media Analysis Engine](https://mlsanity.com/paper/2311.07547/) (2023)
- [Contextualizing Internet Memes Across Social Media Platforms](https://mlsanity.com/paper/2311.11157/) (2023)
- ["Stop Asian Hate!" : Refining Detection of Anti-Asian Hate Speech During the COVID-19 Pandemic](https://mlsanity.com/paper/2112.02265/) (2021)
- [Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection](https://mlsanity.com/paper/2402.09738/) (2024)
- [System Report for CCL25-Eval Task 10: SRAG-MAV for Fine-Grained Chinese Hate Speech Recognition](https://mlsanity.com/paper/2507.18580/) (2025)
- [Detection of Propaganda Techniques in Visuo-Lingual Metaphor in Memes](https://mlsanity.com/paper/2205.02937/) (2022)
- [TOT: Topology-Aware Optimal Transport For Multimodal Hate Detection](https://mlsanity.com/paper/2303.09314/) (2023)
- [Causal Intersectionality and Dual Form of Gradient Descent for Multimodal Analysis: a Case Study on Hateful Memes](https://mlsanity.com/paper/2308.11585/) (2023)
- [Experimentation in Content Moderation using RWKV](https://mlsanity.com/paper/2409.03939/) (2024)
- [MHS-STMA: Multimodal Hate Speech Detection via Scalable Transformer-Based Multilevel Attention Framework](https://mlsanity.com/paper/2409.05136/) (2024)
- [Rethinking Offensive Text Detection as a Multi-Hop Reasoning Problem](https://mlsanity.com/paper/2204.10521/) (2022)

## Abstract (arXiv)

In the past few years, there has been a surge of interest in multi-modal problems, from image captioning to visual question answering and beyond. In this paper, we focus on hate speech detection in multi-modal memes wherein memes pose an interesting multi-modal fusion problem. We aim to solve the Facebook Meme Challenge citekiela2020hateful which aims to solve a binary classification problem of predicting whether a meme is hateful or not. A crucial characteristic of the challenge is that it includes "benign confounders" to counter the possibility of models exploiting unimodal priors. The challenge states that the state-of-the-art models perform poorly compared to humans. During the analysis of the dataset, we realized that majority of the data points which are originally hateful are turned into benign just be describing the image of the meme. Also, majority of the multi-modal baselines give more preference to the hate speech (language modality). To tackle these problems, we explore the visual modality using object detection and image captioning models to fetch the "actual caption" and then combine it with the multi-modal representation to perform binary classification. This approach tackles the benign text confounders present in the dataset to improve the performance. Another approach we experiment with is to improve the prediction with sentiment analysis. Instead of only using multi-modal representations obtained from pre-trained neural networks, we also include the unimodal sentiment to enrich the features. We perform a detailed analysis of the above two approaches, providing compelling reasons in favor of the methodologies used.