mlsanityData to 7 Oct 2026

Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memes

arXiv 2608.26143 ↗First posted 27 Jun 2026cs.CL cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Memes have turned out to be a powerful tool through which individuals share their ideas concerning contemporary social and political problems. Their anonymity, as well as their ability to go viral, make them a powerful medium for spreading hate. It remains very difficult to identify such complex and context-dependent hate speech. Although they display excellent performance on multimodal tasks, vision-language models (VLMs) tend to ignore context, irony, and other subtle cues that play a key role in identifying hateful memes. In this work, we present a qualitative analysis of four state-of-the-art VLMs: LLaVA-7B, Qwen-VL, GPT-4o mini, and Claude 3 Haiku. We evaluate these models under zero-shot and few-shot prompting to examine how contextual framing influences their outputs. Our analysis goes beyond simple classification accuracy and focuses on a qualitative evaluation of the models' generated justifications, providing a more in-depth understanding of their thought processes and constraints when dealing with hateful memes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 52 references that are in the map. First posted on arXiv 27 Jun 2026.

Builds on

28 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models30 Apr

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  10. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  11. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  12. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
15 more
  1. 2020Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challengecited by 51
  2. 2020Detecting Hate Speech in Multi-modal Memescited by 38
  3. 2023Decoding the Underlying Meaning of Multimodal Hateful Memescited by 38
  4. 2024MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classificationcited by 29
  5. 2024MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Interventioncited by 20
  6. 2021Less is More: Lighter and Faster Deep Neural Architecture for Tomato Leaf Disease Classificationcited by 15
  7. 2022Two Decades of Bengali Handwritten Digit Recognition: A Surveycited by 13
  8. 2022Rethinking Cooking State Recognition with Vision Transformerscited by 9
  9. 2025AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditionscited by 9
  10. 2025MemeBLIP2: A novel lightweight multimodal system to detect harmful memescited by 8
  11. 2025Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarizationcited by 6
  12. 2024HaSPeR: An Image Repository for Hand Shadow Puppet Recognitioncited by 5
  13. 2025Performance Analysis of Few-Shot Learning Approaches for Bangla Handwritten Character and Digit Recognitioncited by 4
  14. 2025On VLMs for Diverse Tasks in Multimodal Meme Classificationcited by 2
  15. 2025Preemptive Hallucination Reduction: An Input-Level Approach for Multimodal Language Modelcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretations
  2. 2023Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model
  3. 2026FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detection
  4. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions
  5. 2026From Native Memes to Global Moderation: Cross-Cultural Evaluation of Vision-Language Models for Hateful Meme Detection
  6. 2025Labels or Input? Rethinking Augmentation in Multimodal Hate Detection
  7. 2023A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge
  8. 2025Cross-Modal Transfer from Memes to Videos: Addressing Data Scarcity in Hateful Video Detection