mlsanityData to 7 Oct 2026

Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning

arXiv 2311.08110 ↗First posted 14 Nov 2023cs.CL cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Hateful memes have emerged as a significant concern on the Internet. Detecting hateful memes requires the system to jointly understand the visual and textual modalities. Our investigation reveals that the embedding space of existing CLIP-based systems lacks sensitivity to subtle differences in memes that are vital for correct hatefulness classification. We propose constructing a hatefulness-aware embedding space through retrieval-guided contrastive training. Our approach achieves state-of-the-art performance on the HatefulMemes dataset with an AUROC of 87.0, outperforming much larger fine-tuned large multimodal models. We demonstrate a retrieval-based hateful memes detection system, which is capable of identifying hatefulness based on data unseen in training. This allows developers to update the hateful memes detection system by simply adding new examples without retraining, a desirable feature for real services in the constantly evolving landscape of hateful memes on the Internet.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 29 of its 53 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

29 of 53 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
17 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  3. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  4. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  5. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  6. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  7. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  8. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  9. 2017Smart Mining for Deep Metric Learningcited by 96
  10. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 85
  11. 2020SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!cited by 77
  12. 2021Detecting Harmful Memes and Their Targetscited by 77
  13. 2021Open Domain Question Answering over Tables via Dense Retrievalcited by 74
  14. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  15. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
  16. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  17. 2023TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singaporecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 12.

20 papers.

Sort:

By area:

  1. 2025Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Studycited by 12
  2. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detectioncited by 11
  3. 2025ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detectioncited by 6
  4. 2026MemeLens: Multilingual Multitask VLMs for Memescited by 5
  5. 2025From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignmentcited by 3
  6. 2025Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computingcited by 2
  7. 2025Multi-Agent VLMs Guided Self-Training with PNU Loss for Low-Resource Offensive Content Detectioncited by 2
  8. 2025Ideology-Based LLMs for Content Moderationcited by 1
  9. 2026KID: Knowledge-Injected Dual-Head Learning for Knowledge-Grounded Harmful Meme Detectioncited by 1
  10. 2026Innocent Panels, Hateful Stories: Evaluating and Detecting Hateful Intent in Multi-Turn Visual Story Generationcited by 1
  11. 2026Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classificationcited by 1
  12. 2025Unpacking Hateful Memes: Presupposed Context and False Claimscited by 0
8 more
  1. 2026Fine-grained Verbal Attack Detection via a Hierarchical Divide-and-Conquer Frameworkcited by 0
  2. 2026ExDR: Explanation-driven Dynamic Retrieval Enhancement for Multimodal Fake News Detectioncited by 0
  3. 2026More Than Sum of Its Parts: Deciphering Intent Shifts in Multimodal Hate Speech Detectioncited by 0
  4. 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challengescited by 0
  5. 2026ZeroR@CHiPSAL 2026: Two-Stage Vision-Language Adaptation with Contrastive Learning for Nepali Meme Classificationcited by 0
  6. 2026SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detectioncited by 0
  7. 2026Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detectioncited by 0
  8. 2026Learn Before You Judge: Progressive Knowledge-to-Decision Alignment for Explainable Hateful Meme Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes
  2. 2023Mapping Memes to Words for Multimodal Hateful Meme Classification
  3. 2025Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering
  4. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions
  5. 2025Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge
  6. 2026GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection
  7. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models
  8. 2021Caption Enriched Samples for Improving Hateful Memes Detection