# MemeFier: Dual-stage Modality Fusion for Image Meme Classification

arXiv: https://arxiv.org/abs/2304.02906 · HTML: https://mlsanity.com/paper/2304.02906/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!](https://mlsanity.com/paper/2008.03781/) (2020)
- [Detecting and Understanding Harmful Memes: A Survey](https://mlsanity.com/paper/2205.04274/) (2022)
- [Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features](https://mlsanity.com/paper/2210.05916/) (2022)
- [Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes](https://mlsanity.com/paper/2012.07788/) (2020)
- [Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge](https://mlsanity.com/paper/2012.12975/) (2020)
- [A Multimodal Framework for the Detection of Hateful Memes](https://mlsanity.com/paper/2012.12871/) (2020)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution](https://mlsanity.com/paper/2012.08290/) (2020)
- [Detecting Hateful Memes Using a Multimodal Deep Ensemble](https://mlsanity.com/paper/2012.13235/) (2020)
- [Multimodal Learning for Hateful Memes Detection](https://mlsanity.com/paper/2011.12870/) (2020)
- [Caption Enriched Samples for Improving Hateful Memes Detection](https://mlsanity.com/paper/2109.10649/) (2021)
- [IITK at SemEval-2020 Task 8: Unimodal and Bimodal Sentiment Analysis of Internet Memes](https://mlsanity.com/paper/2007.10822/) (2020)
- [UPB at SemEval-2020 Task 8: Joint Textual and Visual Modeling in a Multi-Task Learning Architecture for Memotion Analysis](https://mlsanity.com/paper/2009.02779/) (2020)
- [Multimodal Analysis of memes for sentiment extraction](https://mlsanity.com/paper/2112.11850/) (2021)
- [NUAA-QMUL at SemEval-2020 Task 8: Utilizing BERT and DenseNet for Internet Meme Emotion Analysis](https://mlsanity.com/paper/2011.02788/) (2020)
- [MemeTector: Enforcing deep focus for meme detection](https://mlsanity.com/paper/2205.13268/) (2022)
- [gundapusunil at SemEval-2020 Task 8: Multimodal Memotion Analysis](https://mlsanity.com/paper/2010.04470/) (2020)
- [Multimodal Feature Extraction for Memes Sentiment Classification](https://mlsanity.com/paper/2207.03317/) (2022)
- [YNU-HPCC at SemEval-2020 Task 8: Using a Parallel-Channel Model for Memotion Analysis](https://mlsanity.com/paper/2007.13968/) (2020)
- [DSC IIT-ISM at SemEval-2020 Task 8: Bi-Fusion Techniques for Deep Meme Emotion Analysis](https://mlsanity.com/paper/2008.00825/) (2020)

## Built on by (most cited first, 6 of 6)

- [RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection](https://mlsanity.com/paper/2311.09939/) (2023)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [Towards Comprehensive Detection of Chinese Harmful Memes](https://mlsanity.com/paper/2410.02378/) (2024)
- [Revealing Temporal Label Noise in Multimodal Hateful Video Classification](https://mlsanity.com/paper/2508.04900/) (2025)
- [Multimodal Hate Detection Using Dual-Stream Graph Neural Networks](https://mlsanity.com/paper/2509.13515/) (2025)
- [MemeTAG: Keyword-Driven Meme Classification through Tag Embedding Reconstruction](https://mlsanity.com/paper/2609.20962/) (2026)

## Abstract (arXiv)

Hate speech is a societal problem that has significantly grown through the Internet. New forms of digital content such as image memes have given rise to spread of hate using multimodal means, being far more difficult to analyse and detect compared to the unimodal case. Accurate automatic processing, analysis and understanding of this kind of content will facilitate the endeavor of hindering hate speech proliferation through the digital world. To this end, we propose MemeFier, a deep learning-based architecture for fine-grained classification of Internet image memes, utilizing a dual-stage modality fusion module. The first fusion stage produces feature vectors containing modality alignment information that captures non-trivial connections between the text and image of a meme. The second fusion stage leverages the power of a Transformer encoder to learn inter-modality correlations at the token level and yield an informative representation. Additionally, we consider external knowledge as an additional input, and background image caption supervision as a regularizing component. Extensive experiments on three widely adopted benchmarks, i.e., Facebook Hateful Memes, Memotion7k and MultiOFF, indicate that our approach competes and in some cases surpasses state-of-the-art. Our code is available on https://github.com/ckoutlis/memefier.