# Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset

arXiv: https://arxiv.org/abs/2107.04313 · HTML: https://mlsanity.com/paper/2107.04313/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings](https://mlsanity.com/paper/1809.02169/) (2018)
- [A Comprehensive Study on Face Recognition Biases Beyond Demographics](https://mlsanity.com/paper/2103.01592/) (2021)
- [Detecting Hate Speech in Multi-modal Memes](https://mlsanity.com/paper/2012.14891/) (2020)
- [Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts](https://mlsanity.com/paper/1904.09073/) (2019)
- [Multimodal Learning for Hateful Memes Detection](https://mlsanity.com/paper/2011.12870/) (2020)
- [Analyzing the hate and counter speech accounts on Twitter](https://mlsanity.com/paper/1812.02712/) (2018)
- [Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge](https://mlsanity.com/paper/2012.01002/) (2020)

## Built on by (most cited first, 13 of 13)

- [Detecting and Understanding Harmful Memes: A Survey](https://mlsanity.com/paper/2205.04274/) (2022)
- [Multi-Granular Multimodal Clue Fusion for Meme Understanding](https://mlsanity.com/paper/2503.12560/) (2025)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretations](https://mlsanity.com/paper/2402.12198/) (2024)
- [MemeCraft: Contextual and Stance-Driven Multimodal Meme Generation](https://mlsanity.com/paper/2403.14652/) (2024)
- [MM-Claims: A Dataset for Multimodal Claim Detection in Social Media](https://mlsanity.com/paper/2205.01989/) (2022)
- [Multimodal and Explainable Internet Meme Classification](https://mlsanity.com/paper/2212.05612/) (2022)
- [Contextualizing Internet Memes Across Social Media Platforms](https://mlsanity.com/paper/2311.11157/) (2023)
- [The State of Profanity Obfuscation in Natural Language Processing](https://mlsanity.com/paper/2210.07595/) (2022)
- [Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering](https://mlsanity.com/paper/2504.16723/) (2025)
- [Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing](https://mlsanity.com/paper/2506.07086/) (2025)
- [Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection](https://mlsanity.com/paper/2609.18860/) (2026)

## Abstract (arXiv)

Hateful memes pose a unique challenge for current machine learning systems because their message is derived from both text- and visual-modalities. To this effect, Facebook released the Hateful Memes Challenge, a dataset of memes with pre-extracted text captions, but it is unclear whether these synthetic examples generalize to `memes in the wild'. In this paper, we collect hateful and non-hateful memes from Pinterest to evaluate out-of-sample performance on models pre-trained on the Facebook dataset. We find that memes in the wild differ in two key aspects: 1) Captions must be extracted via OCR, injecting noise and diminishing performance of multimodal models, and 2) Memes are more diverse than `traditional memes', including screenshots of conversations or text on a plain background. This paper thus serves as a reality check for the current benchmark of hateful meme detection and its applicability for detecting real world hate.