mlsanityData to 7 Oct 2026

Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection

arXiv 2308.08088 ↗First posted 16 Aug 2023cs.CV cs.IR cs.MMcited by 45

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Hateful meme detection is a challenging multimodal task that requires comprehension of both vision and language, as well as cross-modal interactions. Recent studies have tried to fine-tune pre-trained vision-language models (PVLMs) for this task. However, with increasing model sizes, it becomes important to leverage powerful PVLMs more efficiently, rather than simply fine-tuning them. Recently, researchers have attempted to convert meme images into textual captions and prompt language models for predictions. This approach has shown good performance but suffers from non-informative image captions. Considering the two factors mentioned above, we propose a probing-based captioning approach to leverage PVLMs in a zero-shot visual question answering (VQA) manner. Specifically, we prompt a frozen PVLM by asking hateful content-related questions and use the answers as image captions (which we call Pro-Cap), so that the captions contain information critical for hateful content detection. The good performance of models with Pro-Cap on three benchmarks validates the effectiveness and generalization of the proposed method.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 38 references that are in the map. First posted on arXiv 16 Aug 2023.

Builds on

31 of 38 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2017Decoupled Weight Decay Regularizationcited by 1,722
19 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  5. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  6. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  7. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 85
  8. 2021Detecting Harmful Memes and Their Targetscited by 77
  9. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  10. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  11. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
  12. 2020Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challengecited by 51
  13. 2020Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memescited by 50
  14. 2020A Multimodal Framework for the Detection of Hateful Memescited by 48
  15. 2021Disentangling Hate in Online Memescited by 46
  16. 2020Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solutioncited by 41
  17. 2023Decoding the Underlying Meaning of Multimodal Hateful Memescited by 38
  18. 2020Multimodal Learning for Hateful Memes Detectioncited by 29
  19. 2022On Explaining Multimodal Hateful Meme Detection Modelscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 16; 2026: 16.

45 papers.

Sort:

By area:

  1. 2024MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classificationcited by 29
  2. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
  3. 2024MultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibilicited by 25
  4. 2024Towards Low-Resource Harmful Meme Detection with LMM Agentscited by 23
  5. 2024Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Modelscited by 17
  6. 2024Modularized Networks for Few-shot Hateful Meme Detectioncited by 15
  7. 2025MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detectioncited by 14
  8. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  9. 2025ImpliHateVid: A Benchmark Dataset and Two-stage Contrastive Learning Framework for Implicit Hate Speech Detection in Videoscited by 12
  10. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detectioncited by 11
  11. 2024MemeCraft: Contextual and Stance-Driven Multimodal Meme Generationcited by 9
  12. 2024VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuningcited by 9
33 more
  1. 2025Cross-Modal Transfer from Memes to Videos: Addressing Data Scarcity in Hateful Video Detectioncited by 9
  2. 2025MemeIntel: Explainable Detection of Propagandistic and Hateful Memescited by 8
  3. 2024Bridging Modalities: Enhancing Cross-Modality Hate Speech Detection with Few-Shot In-Context Learningcited by 7
  4. 2024Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memescited by 7
  5. 2025Read as You See: Guiding Unimodal LLMs for Low-Resource Explainable Harmful Meme Detectioncited by 6
  6. 2025ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detectioncited by 6
  7. 2025Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comicscited by 5
  8. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memescited by 4
  9. 2025From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignmentcited by 3
  10. 2025Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledgecited by 2
  11. 2026StreamSense: Streaming Social Task Detection with Selective Vision-Language Model Routingcited by 2
  12. 2026Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learningcited by 2
  13. 2026MemeTAG: Keyword-Driven Meme Classification through Tag Embedding Reconstructioncited by 2
  14. 2024Prompt-enhanced Network for Hateful Meme Classificationcited by 1
  15. 2025VModA: An Effective Framework for Adaptive NSFW Image Moderationcited by 1
  16. 2025From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvementcited by 1
  17. 2026All Changes May Have Invariant Principles: Improving Ever-Shifting Harmful Meme Detection via Design Concept Reproductioncited by 1
  18. 2026MEMEWEAVER: Inter-Meme Graph Reasoning for Sexism and Misogyny Detectioncited by 1
  19. 2026KID: Knowledge-Injected Dual-Head Learning for Knowledge-Grounded Harmful Meme Detectioncited by 1
  20. 2026Shedding the Facades, Connecting the Domains: Detecting Shifting Multimodal Hate Video with Test-Time Adaptationcited by 1
  21. 2025STEMTOX: From Collaborative Tags to Fine-Grained Toxic Meme Detection via Entropy-Guided Multi-Task Learningcited by 0
  22. 2025Unpacking Hateful Memes: Presupposed Context and False Claimscited by 0
  23. 2025MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Modelscited by 0
  24. 2025Malicious Image Analysis via Vision-Language Segmentation Fusion: Detection, Element, and Location in One-shotcited by 0
  25. 2026More Than Sum of Its Parts: Deciphering Intent Shifts in Multimodal Hate Speech Detectioncited by 0
  26. 2026PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Frameworkcited by 0
  27. 2026QwenSafe: Multimodal Content Rating Description Identification via Preference-Aligned VLMscited by 0
  28. 2026I Know What You Meme, Even If it Emerged Today: Understanding Evolving Memes through Open-World Knowledge Acquisitioncited by 0
  29. 2026AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memescited by 0
  30. 2026Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memescited by 0
  31. 2026Towards High-Level Semantic Intelligencecited by 0
  32. 2026CLARA: Clip-Level Multimodal Alignment with VLM-Derived Rationales for Hateful Video Detectioncited by 0
  33. 2026Through the Eyes of the Beholder: Biometric and Demographic Conditioning for Multimodal Sexism Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Caption Enriched Samples for Improving Hateful Memes Detection
  2. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models
  3. 2024Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretations
  4. 2023Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model
  5. 2025Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering
  6. 2026Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memes
  7. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions
  8. 2026MemeScouts@LT-EDI 2026: Asking the Right Questions -- Prompted Weak Supervision for Meme Hate Speech Detection