# Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memes

arXiv: https://arxiv.org/abs/2608.26143 · HTML: https://mlsanity.com/paper/2608.26143/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 52 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions](https://mlsanity.com/paper/2404.07214/) (2024)
- [Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features](https://mlsanity.com/paper/2210.05916/) (2022)
- [Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge](https://mlsanity.com/paper/2012.12975/) (2020)
- [Decoding the Underlying Meaning of Multimodal Hateful Memes](https://mlsanity.com/paper/2305.17678/) (2023)
- [Detecting Hate Speech in Multi-modal Memes](https://mlsanity.com/paper/2012.14891/) (2020)
- [MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classification](https://mlsanity.com/paper/2409.14703/) (2024)
- [MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Intervention](https://mlsanity.com/paper/2406.05344/) (2024)
- [Less is More: Lighter and Faster Deep Neural Architecture for Tomato Leaf Disease Classification](https://mlsanity.com/paper/2109.02394/) (2021)
- [Two Decades of Bengali Handwritten Digit Recognition: A Survey](https://mlsanity.com/paper/2206.02234/) (2022)
- [Rethinking Cooking State Recognition with Vision Transformers](https://mlsanity.com/paper/2212.08586/) (2022)
- [AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](https://mlsanity.com/paper/2506.17455/) (2025)
- [MemeBLIP2: A novel lightweight multimodal system to detect harmful memes](https://mlsanity.com/paper/2504.21226/) (2025)
- [Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarization](https://mlsanity.com/paper/2505.05070/) (2025)
- [HaSPeR: An Image Repository for Hand Shadow Puppet Recognition](https://mlsanity.com/paper/2408.10360/) (2024)
- [Performance Analysis of Few-Shot Learning Approaches for Bangla Handwritten Character and Digit Recognition](https://mlsanity.com/paper/2506.00447/) (2025)
- [Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models](https://mlsanity.com/paper/2505.00150/) (2025)
- [On VLMs for Diverse Tasks in Multimodal Meme Classification](https://mlsanity.com/paper/2505.20937/) (2025)
- [Preemptive Hallucination Reduction: An Input-Level Approach for Multimodal Language Model](https://mlsanity.com/paper/2505.24007/) (2025)

## Abstract (arXiv)

Memes have turned out to be a powerful tool through which individuals share their ideas concerning contemporary social and political problems. Their anonymity, as well as their ability to go viral, make them a powerful medium for spreading hate. It remains very difficult to identify such complex and context-dependent hate speech. Although they display excellent performance on multimodal tasks, vision-language models (VLMs) tend to ignore context, irony, and other subtle cues that play a key role in identifying hateful memes. In this work, we present a qualitative analysis of four state-of-the-art VLMs: LLaVA-7B, Qwen-VL, GPT-4o mini, and Claude 3 Haiku. We evaluate these models under zero-shot and few-shot prompting to examine how contextual framing influences their outputs. Our analysis goes beyond simple classification accuracy and focuses on a qualitative evaluation of the models' generated justifications, providing a more in-depth understanding of their thought processes and constraints when dealing with hateful memes.