# Hateful Memes Detection via Complementary Visual and Linguistic Networks

arXiv: https://arxiv.org/abs/2012.04977 · HTML: https://mlsanity.com/paper/2012.04977/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 7 of 7)

- [Prompting for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2302.04156/) (2023)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [Decoding the Underlying Meaning of Multimodal Hateful Memes](https://mlsanity.com/paper/2305.17678/) (2023)
- [Detecting Harmful Content On Online Platforms: What Platforms Need Vs. Where Research Efforts Go](https://mlsanity.com/paper/2103.00153/) (2021)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [MM-HSD: Multi-Modal Hate Speech Detection in Videos](https://mlsanity.com/paper/2508.20546/) (2025)
- [Unpacking Hateful Memes: Presupposed Context and False Claims](https://mlsanity.com/paper/2510.09935/) (2025)

## Abstract (arXiv)

Hateful memes are widespread in social media and convey negative information. The main challenge of hateful memes detection is that the expressive meaning can not be well recognized by a single modality. In order to further integrate modal information, we investigate a candidate solution based on complementary visual and linguistic network in Hateful Memes Challenge 2020. In this way, more comprehensive information of the multi-modality could be explored in detail. Both contextual-level and sensitive object-level information are considered in visual and linguistic embedding to formulate the complex multi-modal scenarios. Specifically, a pre-trained classifier and object detector are utilized to obtain the contextual features and region-of-interests (RoIs) from the input, followed by the position representation fusion for visual embedding. While linguistic embedding is composed of three components, i.e., the sentence words embedding, position embedding and the corresponding Spacy embedding (Sembedding), which is a symbol represented by vocabulary extracted by Spacy. Both visual and linguistic embedding are fed into the designed Complementary Visual and Linguistic (CVL) networks to produce the prediction for hateful memes. Experimental results on Hateful Memes Challenge Dataset demonstrate that CVL provides a decent performance, and produces 78:48% and 72:95% on the criteria of AUROC and Accuracy. Code is available at https://github.com/webYFDT/hateful.