# Disentangling Hate in Online Memes

arXiv: https://arxiv.org/abs/2108.06207 · HTML: https://mlsanity.com/paper/2108.06207/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 60 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Large-Scale Adversarial Training for Vision-and-Language Representation Learning](https://mlsanity.com/paper/2006.06195/) (2020)
- [ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph](https://mlsanity.com/paper/2006.16934/) (2020)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [Learning Disentangled Representations for Recommendation](https://mlsanity.com/paper/1910.14238/) (2019)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations](https://mlsanity.com/paper/1705.08841/) (2017)
- [FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age](https://mlsanity.com/paper/1908.04913/) (2019)
- [Learning Disentangled Joint Continuous and Discrete Representations](https://mlsanity.com/paper/1804.00104/) (2018)
- [All You Need is "Love": Evading Hate-speech Detection](https://mlsanity.com/paper/1808.09115/) (2018)
- [Learning Disentangled Representations via Mutual Information Estimation](https://mlsanity.com/paper/1912.03915/) (2019)
- [Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes](https://mlsanity.com/paper/2012.07788/) (2020)
- [Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge](https://mlsanity.com/paper/2012.12975/) (2020)
- [A Multimodal Framework for the Detection of Hateful Memes](https://mlsanity.com/paper/2012.12871/) (2020)
- [One-step and Two-step Classification for Abusive Language Detection on Twitter](https://mlsanity.com/paper/1706.01206/) (2017)
- [Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution](https://mlsanity.com/paper/2012.08290/) (2020)
- [Detecting Hate Speech in Multi-modal Memes](https://mlsanity.com/paper/2012.14891/) (2020)

## Built on by (most cited first, 30 of 46)

- [Prompting for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2302.04156/) (2023)
- [Detecting and Understanding Harmful Memes: A Survey](https://mlsanity.com/paper/2205.04274/) (2022)
- [Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection](https://mlsanity.com/paper/2308.08088/) (2023)
- [Decoding the Underlying Meaning of Multimodal Hateful Memes](https://mlsanity.com/paper/2305.17678/) (2023)
- [On Explaining Multimodal Hateful Meme Detection Models](https://mlsanity.com/paper/2204.01734/) (2022)
- [Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models](https://mlsanity.com/paper/2401.16727/) (2024)
- [Towards Low-Resource Harmful Meme Detection with LMM Agents](https://mlsanity.com/paper/2411.05383/) (2024)
- [Mapping Memes to Words for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2310.08368/) (2023)
- [BanglaAbuseMeme: A Dataset for Bengali Abusive Meme Classification](https://mlsanity.com/paper/2310.11748/) (2023)
- [Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models](https://mlsanity.com/paper/2401.13298/) (2024)
- [Modularized Networks for Few-shot Hateful Meme Detection](https://mlsanity.com/paper/2402.11845/) (2024)
- [MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection](https://mlsanity.com/paper/2507.06908/) (2025)
- [Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions](https://mlsanity.com/paper/2502.11073/) (2025)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singapore](https://mlsanity.com/paper/2305.17911/) (2023)
- [Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modality](https://mlsanity.com/paper/2305.10547/) (2023)
- [MemeCraft: Contextual and Stance-Driven Multimodal Meme Generation](https://mlsanity.com/paper/2403.14652/) (2024)
- [Predicting Hate Intensity of Twitter Conversation Threads](https://mlsanity.com/paper/2206.08406/) (2022)
- [Deciphering Hate: Identifying Hateful Memes and Their Targets](https://mlsanity.com/paper/2403.10829/) (2024)
- [Hate Speech Detection with Generalizable Target-aware Fairness](https://mlsanity.com/paper/2406.00046/) (2024)
- [Bridging Modalities: Enhancing Cross-Modality Hate Speech Detection with Few-Shot In-Context Learning](https://mlsanity.com/paper/2410.05600/) (2024)
- [MemeFier: Dual-stage Modality Fusion for Image Meme Classification](https://mlsanity.com/paper/2304.02906/) (2023)
- [MemeGraphs: Linking Memes to Knowledge Graphs](https://mlsanity.com/paper/2305.18391/) (2023)
- [PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models](https://mlsanity.com/paper/2312.06093/) (2023)
- [LionGuard: Building a Contextualized Moderation Classifier to Tackle Localized Unsafe Content](https://mlsanity.com/paper/2407.10995/) (2024)
- [Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection](https://mlsanity.com/paper/2504.20644/) (2025)
- [Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics](https://mlsanity.com/paper/2509.12248/) (2025)
- [Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection](https://mlsanity.com/paper/2402.09738/) (2024)
- [A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension](https://mlsanity.com/paper/2508.16300/) (2025)
- [TOT: Topology-Aware Optimal Transport For Multimodal Hate Detection](https://mlsanity.com/paper/2303.09314/) (2023)

## Abstract (arXiv)

Hateful and offensive content detection has been extensively explored in a single modality such as text. However, such toxic information could also be communicated via multimodal content such as online memes. Therefore, detecting multimodal hateful content has recently garnered much attention in academic and industry research communities. This paper aims to contribute to this emerging research topic by proposing DisMultiHate, which is a novel framework that performed the classification of multimodal hateful content. Specifically, DisMultiHate is designed to disentangle target entities in multimodal memes to improve hateful content classification and explainability. We conduct extensive experiments on two publicly available hateful and offensive memes datasets. Our experiment results show that DisMultiHate is able to outperform state-of-the-art unimodal and multimodal baselines in the hateful meme classification task. Empirical case studies were also conducted to demonstrate DisMultiHate's ability to disentangle target entities in memes and ultimately showcase DisMultiHate's explainability of the multimodal hateful content classification task.