mlsanityData to 7 Oct 2026

HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes

arXiv 2408.05794 ↗First posted 11 Aug 2024cs.AI cs.CL cs.MMcited by 2

Not on the map. Why?

Abstract (arXiv)

Amidst the rise of Large Multimodal Models (LMMs) and their widespread application in generating and interpreting complex content, the risk of propagating biased and harmful memes remains significant. Current safety measures often fail to detect subtly integrated hateful content within ``Confounder Memes''. To address this, we introduce textscHateSieve, a new framework designed to enhance the detection and segmentation of hateful elements in memes. textscHateSieve features a novel Contrastive Meme Generator that creates semantically paired memes, a customized triplet dataset for contrastive learning, and an Image-Text Alignment module that produces context-aware embeddings for accurate meme segmentation. Empirical experiments on the Hateful Meme Dataset show that textscHateSieve not only surpasses existing LMMs in performance with fewer trainable parameters but also offers a robust mechanism for precisely identifying and isolating hateful content. textcolorredCaution: Contains academic discussions of hate speech; viewer discretion advised.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 11 Aug 2024.

Builds on

30 of 46 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Segment Anythingcited by 8,158
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2017Mask R-CNNcited by 7,259
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
18 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  2. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  3. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  4. 2023Detecting Language Model Attacks with Perplexitycited by 411
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  6. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
  7. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  8. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 161
  9. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  10. 2019Exploring Hate Speech Detection in Multimodal Publicationscited by 103
  11. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 85
  12. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  13. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
  14. 2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 52
  15. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 50
  16. 2023Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Informationcited by 32
  17. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biasescited by 13
  18. 2023AI in the Gray: Exploring Moderation Policies in Dialogic Large Language Models vs. Human Answers in Controversial Topicscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detectioncited by 6
  2. 2026Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning
  2. 2023Mapping Memes to Words for Multimodal Hateful Meme Classification
  3. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions
  4. 2026GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection
  5. 2025Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge
  6. 2025Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering
  7. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models
  8. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection