# Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggers

arXiv: https://arxiv.org/abs/2412.15503 · HTML: https://mlsanity.com/paper/2412.15503/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2024.

## Built on by (most cited first, 2 of 2)

- [SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection](https://mlsanity.com/paper/2509.16060/) (2025)
- [MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models](https://mlsanity.com/paper/2510.27196/) (2025)

## Abstract (arXiv)

Hateful meme detection aims to prevent the proliferation of hateful memes on various social media platforms. Considering its impact on social environments, this paper introduces a previously ignored but significant threat to hateful meme detection: backdoor attacks. By injecting specific triggers into meme samples, backdoor attackers can manipulate the detector to output their desired outcomes. To explore this, we propose the Meme Trojan framework to initiate backdoor attacks on hateful meme detection. Meme Trojan involves creating a novel Cross-Modal Trigger (CMT) and a learnable trigger augmentor to enhance the trigger pattern according to each input sample. Due to the cross-modal property, the proposed CMT can effectively initiate backdoor attacks on hateful meme detectors under an automatic application scenario. Additionally, the injection position and size of our triggers are adaptive to the texts contained in the meme, which ensures that the trigger is seamlessly integrated with the meme content. Our approach outperforms the state-of-the-art backdoor attack methods, showing significant improvements in effectiveness and stealthiness. We believe that this paper will draw more attention to the potential threat posed by backdoor attacks on hateful meme detection.