mlsanityData to 7 Oct 2026

Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions

arXiv 2502.11073 ↗First posted 16 Feb 2025cs.CLcited by 13

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Hateful meme detection presents a significant challenge as a multimodal task due to the complexity of interpreting implicit hate messages and contextual cues within memes. Previous approaches have fine-tuned pre-trained vision-language models (PT-VLMs), leveraging the knowledge they gained during pre-training and their attention mechanisms to understand meme content. However, the reliance of these models on implicit knowledge and complex attention mechanisms renders their decisions difficult to explain, which is crucial for building trust in meme classification. In this paper, we introduce IntMeme, a novel framework that leverages Large Multimodal Models (LMMs) for hateful meme classification with explainable decisions. IntMeme addresses the dual challenges of improving both accuracy and explainability in meme moderation. The framework uses LMMs to generate human-like, interpretive analyses of memes, providing deeper insights into multimodal content and context. Additionally, it uses independent encoding modules for both memes and their interpretations, which are then combined to enhance classification performance. Our approach addresses the opacity and misclassification issues associated with PT-VLMs, optimizing the use of LMMs for hateful meme detection. We demonstrate the effectiveness of IntMeme through comprehensive experiments across three datasets, showcasing its superiority over state-of-the-art models.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 75 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

35 of 75 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  12. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
23 more
  1. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  3. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  4. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  5. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  6. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  7. 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 265
  8. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  9. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  10. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  11. 2020Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challengecited by 51
  12. 2022Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Modelscited by 51
  13. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 50
  14. 2021Disentangling Hate in Online Memescited by 46
  15. 2023Evaluating GPT-3 Generated Explanations for Hateful Content Moderationcited by 44
  16. 2023Decoding the Underlying Meaning of Multimodal Hateful Memescited by 38
  17. 2022On Explaining Multimodal Hateful Meme Detection Modelscited by 29
  18. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
  19. 2022How Hate Speech Varies by Target Identity: A Computational Analysiscited by 23
  20. 2024Modularized Networks for Few-shot Hateful Meme Detectioncited by 15
  21. 2024SGHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Singaporecited by 10
  22. 2024Bridging Modalities: Enhancing Cross-Modality Hate Speech Detection with Few-Shot In-Context Learningcited by 7
  23. 2022Multimodal and Explainable Internet Meme Classificationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025Read as You See: Guiding Unimodal LLMs for Low-Resource Explainable Harmful Meme Detectioncited by 6
  2. 2025ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detectioncited by 6
  3. 2025Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comicscited by 5
  4. 2026MemeLens: Multilingual Multitask VLMs for Memescited by 5
  5. 2026PEST: Parameter Efficient Steering of Blackbox VLMs via Agentic Few-shot Alignment for Hateful Meme Moderationcited by 4
  6. 2025Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languagescited by 2
  7. 2025Is GPT-4o mini Blinded by its Own Safety Filters? Exposing the Multimodal-to-Unimodal Bottleneck in Hate Speech Detectioncited by 0
  8. 2025Reasoning-Aware Multimodal Fusion for Hateful Video Detectioncited by 0
  9. 2026PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Frameworkcited by 0
  10. 2026MuPHI: Learning Implicit Multimodal Harm Reasoning via Semantically Grounded Reward Optimizationcited by 0
  11. 2026FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detectioncited by 0
  12. 2026Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memescited by 0
1 more
  1. 2026Learn Before You Judge: Progressive Knowledge-to-Decision Alignment for Explainable Hateful Meme Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge
  2. 2023Mapping Memes to Words for Multimodal Hateful Meme Classification
  3. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models
  4. 2023A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge
  5. 2024HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes
  6. 2025Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answering
  7. 2026Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memes
  8. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection