# Multimodal and Explainable Internet Meme Classification

arXiv: https://arxiv.org/abs/2212.05612 · HTML: https://mlsanity.com/paper/2212.05612/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 11 Dec 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Understanding Abuse: A Typology of Abusive Language Detection Subtasks](https://mlsanity.com/paper/1705.09899/) (2017)
- [The spread of low-credibility content by social bots](https://mlsanity.com/paper/1707.07592/) (2017)
- [Towards Explainable Deep Neural Networks (xDNN)](https://mlsanity.com/paper/1912.02523/) (2019)
- ["Call me sexist, but...": Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples](https://mlsanity.com/paper/2004.12764/) (2020)
- [Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaign](https://mlsanity.com/paper/1802.04291/) (2018)
- [Disinformation Warfare: Understanding State-Sponsored Trolls on Twitter and Their Influence on the Web](https://mlsanity.com/paper/1801.09288/) (2018)
- [Online misinformation is linked to early COVID-19 vaccination hesitancy and refusal](https://mlsanity.com/paper/2104.10635/) (2021)
- [Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset](https://mlsanity.com/paper/2107.04313/) (2021)
- [Dissecting the Meme Magic: Understanding Indicators of Virality in Image Memes](https://mlsanity.com/paper/2101.06535/) (2021)
- [Who Let The Trolls Out? Towards Understanding State-Sponsored Trolls](https://mlsanity.com/paper/1811.03130/) (2018)
- [Red Bots Do It Better: Comparative Analysis of Social Bot Partisan Behavior](https://mlsanity.com/paper/1902.02765/) (2019)
- [A large-scale crowdsourced analysis of abuse against women journalists and politicians on Twitter](https://mlsanity.com/paper/1902.03093/) (2019)
- [Are Anti-Feminist Communities Gateways to the Far Right? Evidence from Reddit and YouTube](https://mlsanity.com/paper/2102.12837/) (2021)
- [Play MNIST For Me! User Studies on the Effects of Post-Hoc, Example-Based Explanations & Error Rates on Debugging a Deep Learning, Black-Box Classifier](https://mlsanity.com/paper/2009.06349/) (2020)

## Built on by (most cited first, 5 of 5)

- [Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions](https://mlsanity.com/paper/2502.11073/) (2025)
- [MemeCraft: Contextual and Stance-Driven Multimodal Meme Generation](https://mlsanity.com/paper/2403.14652/) (2024)
- [Enhancing Concept Localization in CLIP-based Concept Bottleneck Models](https://mlsanity.com/paper/2510.07115/) (2025)
- [Contextualizing Internet Memes Across Social Media Platforms](https://mlsanity.com/paper/2311.11157/) (2023)
- [Token-Region Guided Cross-Attention Fusion for Multimodal Affect Interpretation](https://mlsanity.com/paper/2607.23493/) (2026)

## Abstract (arXiv)

In the current context where online platforms have been effectively weaponized in a variety of geo-political events and social issues, Internet memes make fair content moderation at scale even more difficult. Existing work on meme classification and tracking has focused on black-box methods that do not explicitly consider the semantics of the memes or the context of their creation. In this paper, we pursue a modular and explainable architecture for Internet meme understanding. We design and implement multimodal classification methods that perform example- and prototype-based reasoning over training cases, while leveraging both textual and visual SOTA models to represent the individual cases. We study the relevance of our modular and explainable models in detecting harmful memes on two existing tasks: Hate Speech Detection and Misogyny Classification. We compare the performance between example- and prototype-based methods, and between text, vision, and multimodal models, across different categories of harmfulness (e.g., stereotype and objectification). We devise a user-friendly interface that facilitates the comparative analysis of examples retrieved by all of our models for any given meme, informing the community about the strengths and limitations of these explainable methods.