mlsanityData to 7 Oct 2026

Mapping Memes to Words for Multimodal Hateful Meme Classification

arXiv 2310.08368 ↗First posted 12 Oct 2023cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Multimodal image-text memes are prevalent on the internet, serving as a unique form of communication that combines visual and textual elements to convey humor, ideas, or emotions. However, some memes take a malicious turn, promoting hateful content and perpetuating discrimination. Detecting hateful memes within this multimodal context is a challenging task that requires understanding the intertwined meaning of text and images. In this work, we address this issue by proposing a novel approach named ISSUES for multimodal hateful meme classification. ISSUES leverages a pre-trained CLIP vision-language model and the textual inversion technique to effectively capture the multimodal semantic content of the memes. The experiments show that our method achieves state-of-the-art results on the Hateful Memes Challenge and HarMeme datasets. The code and the pre-trained models are publicly available at https://github.com/miccunifi/ISSUES.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 20 of its 25 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

20 of 25 references.

References · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  6. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  7. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  8. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  9. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 155
  10. 2023LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-Oncited by 151
  11. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  12. 2021Detecting Harmful Memes and Their Targetscited by 77
8 more
  1. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  2. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
  3. 2020Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challengecited by 51
  4. 2020Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memescited by 50
  5. 2020A Multimodal Framework for the Detection of Hateful Memescited by 48
  6. 2021Disentangling Hate in Online Memescited by 46
  7. 2020Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solutioncited by 41
  8. 2020Detecting Hateful Memes Using a Multimodal Deep Ensemblecited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 7.

19 papers.

Sort:

By area:

  1. 2024MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classificationcited by 29
  2. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  3. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detectioncited by 11
  4. 2025Towards a Robust Framework for Multimodal Hate Detection: A Study on Video vs. Image-based Contentcited by 9
  5. 2025MemeIntel: Explainable Detection of Propagandistic and Hateful Memescited by 8
  6. 2025Read as You See: Guiding Unimodal LLMs for Low-Resource Explainable Harmful Meme Detectioncited by 6
  7. 2025ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detectioncited by 6
  8. 2025From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignmentcited by 3
  9. 2025Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledgecited by 2
  10. 2026MemeTAG: Keyword-Driven Meme Classification through Tag Embedding Reconstructioncited by 2
  11. 2024SLANT: Spurious Logo ANalysis Toolkitcited by 1
  12. 2025Meme Similarity and Emotion Detection using Multimodal Analysiscited by 1
7 more
  1. 2026Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classificationcited by 1
  2. 2025Unpacking Hateful Memes: Presupposed Context and False Claimscited by 0
  3. 2026DARC-CLIP: Dynamic Adaptive Refinement with Cross-Attention for Meme Understandingcited by 0
  4. 2026MuPHI: Learning Implicit Multimodal Harm Reasoning via Semantically Grounded Reward Optimizationcited by 0
  5. 2026Token-Region Guided Cross-Attention Fusion for Multimodal Affect Interpretationcited by 0
  6. 2026CLARA: Clip-Level Multimodal Alignment with VLM-Derived Rationales for Hateful Video Detectioncited by 0
  7. 2026Learn Before You Judge: Progressive Knowledge-to-Decision Alignment for Explainable Hateful Meme Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions
  2. 2025Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering
  3. 2024HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes
  4. 2021Caption Enriched Samples for Improving Hateful Memes Detection
  5. 2023A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge
  6. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models
  7. 2023Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning
  8. 2026GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection