# Multimodal Learning for Hateful Memes Detection

arXiv: https://arxiv.org/abs/2011.12870 · HTML: https://mlsanity.com/paper/2011.12870/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2020.

## Built on by (most cited first, 29 of 29)

- [MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2109.05184/) (2021)
- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [Prompting for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2302.04156/) (2023)
- [Detecting and Understanding Harmful Memes: A Survey](https://mlsanity.com/paper/2205.04274/) (2022)
- [Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection](https://mlsanity.com/paper/2308.08088/) (2023)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [On Explaining Multimodal Hateful Meme Detection Models](https://mlsanity.com/paper/2204.01734/) (2022)
- [Towards Low-Resource Harmful Meme Detection with LMM Agents](https://mlsanity.com/paper/2411.05383/) (2024)
- [DISARM: Detecting the Victims Targeted by Harmful Memes](https://mlsanity.com/paper/2205.05738/) (2022)
- [What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memes](https://mlsanity.com/paper/2212.00715/) (2022)
- [Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models](https://mlsanity.com/paper/2401.13298/) (2024)
- [MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection](https://mlsanity.com/paper/2507.06908/) (2025)
- [Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?](https://mlsanity.com/paper/2301.11219/) (2023)
- [Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset](https://mlsanity.com/paper/2107.04313/) (2021)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing](https://mlsanity.com/paper/2405.11215/) (2024)
- [MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization](https://mlsanity.com/paper/2305.15913/) (2023)
- [Deciphering Hate: Identifying Hateful Memes and Their Targets](https://mlsanity.com/paper/2403.10829/) (2024)
- [D-HUMOR: Dark Humor Understanding via Multimodal Open-ended Reasoning -- A Benchmark Dataset and Method](https://mlsanity.com/paper/2509.06771/) (2025)
- [MemeFier: Dual-stage Modality Fusion for Image Meme Classification](https://mlsanity.com/paper/2304.02906/) (2023)
- [MemeTector: Enforcing deep focus for meme detection](https://mlsanity.com/paper/2205.13268/) (2022)
- [SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memes](https://mlsanity.com/paper/2412.20541/) (2024)
- [TOT: Topology-Aware Optimal Transport For Multimodal Hate Detection](https://mlsanity.com/paper/2303.09314/) (2023)
- [Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing](https://mlsanity.com/paper/2506.07086/) (2025)
- [Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning](https://mlsanity.com/paper/2602.01101/) (2026)
- [Prompt-enhanced Network for Hateful Meme Classification](https://mlsanity.com/paper/2411.07527/) (2024)
- [Decoding Memes: Benchmarking Narrative Role Classification across Multilingual and Multimodal Models](https://mlsanity.com/paper/2506.23122/) (2025)
- [Unpacking Hateful Memes: Presupposed Context and False Claims](https://mlsanity.com/paper/2510.09935/) (2025)
- [Detecting Hate and Inflammatory Content in Bengali Memes: A New Multimodal Dataset and Co-Attention Framework](https://mlsanity.com/paper/2602.22391/) (2026)

## Abstract (arXiv)

Memes are used for spreading ideas through social networks. Although most memes are created for humor, some memes become hateful under the combination of pictures and text. Automatically detecting the hateful memes can help reduce their harmful social impact. Unlike the conventional multimodal tasks, where the visual and textual information is semantically aligned, the challenge of hateful memes detection lies in its unique multimodal information. The image and text in memes are weakly aligned or even irrelevant, which requires the model to understand the content and perform reasoning over multiple modalities. In this paper, we focus on multimodal hateful memes detection and propose a novel method that incorporates the image captioning process into the memes detection process. We conduct extensive experiments on multimodal meme datasets and illustrated the effectiveness of our approach. Our model achieves promising results on the Hateful Memes Detection Challenge.