# Detecting Hateful Memes Using a Multimodal Deep Ensemble

arXiv: https://arxiv.org/abs/2012.13235 · HTML: https://mlsanity.com/paper/2012.13235/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2020.

## Built on by (most cited first, 30 of 30)

- [MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2109.05184/) (2021)
- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [Prompting for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2302.04156/) (2023)
- [Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features](https://mlsanity.com/paper/2210.05916/) (2022)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models](https://mlsanity.com/paper/2401.16727/) (2024)
- [Towards Low-Resource Harmful Meme Detection with LMM Agents](https://mlsanity.com/paper/2411.05383/) (2024)
- [DISARM: Detecting the Victims Targeted by Harmful Memes](https://mlsanity.com/paper/2205.05738/) (2022)
- [Mapping Memes to Words for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2310.08368/) (2023)
- [What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memes](https://mlsanity.com/paper/2212.00715/) (2022)
- [Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models](https://mlsanity.com/paper/2401.13298/) (2024)
- [Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models](https://mlsanity.com/paper/2312.05434/) (2023)
- [MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection](https://mlsanity.com/paper/2507.06908/) (2025)
- [Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?](https://mlsanity.com/paper/2301.11219/) (2023)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model](https://mlsanity.com/paper/2311.06737/) (2023)
- [MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing](https://mlsanity.com/paper/2405.11215/) (2024)
- [MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization](https://mlsanity.com/paper/2305.15913/) (2023)
- [MemeFier: Dual-stage Modality Fusion for Image Meme Classification](https://mlsanity.com/paper/2304.02906/) (2023)
- [MM-HSD: Multi-Modal Hate Speech Detection in Videos](https://mlsanity.com/paper/2508.20546/) (2025)
- [SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memes](https://mlsanity.com/paper/2412.20541/) (2024)
- [An Interpretable Approach to Hateful Meme Detection](https://mlsanity.com/paper/2108.10069/) (2021)
- [Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models](https://mlsanity.com/paper/2505.00150/) (2025)
- [Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing](https://mlsanity.com/paper/2506.07086/) (2025)
- [Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning](https://mlsanity.com/paper/2602.01101/) (2026)
- [Feels Bad Man: Dissecting Automated Hateful Meme Detection Through the Lens of Facebook's Challenge](https://mlsanity.com/paper/2202.08492/) (2022)
- [COT: A Generative Approach for Hate Speech Counter-Narratives via Contrastive Optimal Transport](https://mlsanity.com/paper/2406.12304/) (2024)
- [LIIR at SemEval-2021 task 6: Detection of Persuasion Techniques In Texts and Images using CLIP features](https://mlsanity.com/paper/2105.14774/) (2021)
- [MATK: The Meme Analytical Tool Kit](https://mlsanity.com/paper/2312.06094/) (2023)
- [Fall into a Pit, Gain in a Wit: Cognitive-Guided Harmful Meme Detection via Misjudgment Risk Pattern Retrieval](https://mlsanity.com/paper/2510.15946/) (2025)

## Abstract (arXiv)

While significant progress has been made using machine learning algorithms to detect hate speech, important technical challenges still remain to be solved in order to bring their performance closer to human accuracy. We investigate several of the most recent visual-linguistic Transformer architectures and propose improvements to increase their performance for this task. The proposed model outperforms the baselines by a large margin and ranks 5th on the leaderboard out of 3,100+ participants.