mlsanityData to 7 Oct 2026

On Explaining Multimodal Hateful Meme Detection Models

arXiv 2204.01734 ↗First posted 4 Apr 2022cs.CV cs.SIcited by 29

Not on the map. Why?

Abstract (arXiv)

Hateful meme detection is a new multimodal task that has gained significant traction in academic and industry research communities. Recently, researchers have applied pre-trained visual-linguistic models to perform the multimodal classification task, and some of these solutions have yielded promising results. However, what these visual-linguistic models learn for the hateful meme classification task remains unclear. For instance, it is unclear if these models are able to capture the derogatory or slurs references in multimodality (i.e., image and text) of the hateful memes. To fill this research gap, this paper propose three research questions to improve our understanding of these visual-linguistic models performing the hateful meme classification task. We found that the image modality contributes more to the hateful meme classification task, and the visual-linguistic models are able to perform visual-text slurs grounding to a certain extent. Our error analysis also shows that the visual-linguistic models have acquired biases, which resulted in false-positive predictions.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 4 Apr 2022.

Builds on

31 of 36 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
19 more
  1. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  2. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  3. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  4. 2019Exploring Hate Speech Detection in Multimodal Publicationscited by 103
  5. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  6. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  7. 2020Contextualizing Hate Speech Classifiers with Post-hoc Explanationcited by 84
  8. 2021Detecting Harmful Memes and Their Targetscited by 77
  9. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  10. 2020Demoting Racial Bias in Hate Speech Detectioncited by 62
  11. 2020Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challengecited by 51
  12. 2020A Multimodal Framework for the Detection of Hateful Memescited by 48
  13. 2021Disentangling Hate in Online Memescited by 46
  14. 2020Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solutioncited by 41
  15. 2020Stereotypical Bias Removal for Hate Speech Detection Task using Knowledge-based Generalizationscited by 40
  16. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34
  17. 2020Multimodal Learning for Hateful Memes Detectioncited by 29
  18. 2016Towards Transparent AI Systems: Interpreting Visual Question Answering Modelscited by 19
  19. 2020Robust Explanations for Visual Question Answeringcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 10; 2025: 7; 2026: 4.

29 papers.

Sort:

By area:

  1. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  2. 2023Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detectioncited by 45
  3. 2023Decoding the Underlying Meaning of Multimodal Hateful Memescited by 38
  4. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
  5. 2024Towards Low-Resource Harmful Meme Detection with LMM Agentscited by 23
  6. 2024Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Modelscited by 17
  7. 2025MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detectioncited by 14
  8. 2024Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanationscited by 13
  9. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  10. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisionscited by 13
  11. 2023TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singaporecited by 12
  12. 2024Towards Comprehensive Detection of Chinese Harmful Memescited by 12
17 more
  1. 2024MemeCraft: Contextual and Stance-Driven Multimodal Meme Generationcited by 9
  2. 2024Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detectioncited by 8
  3. 2025MemeIntel: Explainable Detection of Propagandistic and Hateful Memescited by 8
  4. 2023HateProof: Are Hateful Meme Detection Systems really Robust?cited by 7
  5. 2024Bridging Modalities: Enhancing Cross-Modality Hate Speech Detection with Few-Shot In-Context Learningcited by 7
  6. 2023PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Modelscited by 5
  7. 2025MemeReaCon: Probing Contextual Meme Understanding in Large Vision-Language Modelscited by 4
  8. 2026PEST: Parameter Efficient Steering of Blackbox VLMs via Agentic Few-shot Alignment for Hateful Meme Moderationcited by 4
  9. 2024Multimodal Contrastive In-Context Learningcited by 3
  10. 2025From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignmentcited by 3
  11. 2023Follow Us and Become Famous! Insights and Guidelines From Instagram Engagement Mechanismscited by 2
  12. 2023Causal Intersectionality and Dual Form of Gradient Descent for Multimodal Analysis: a Case Study on Hateful Memescited by 2
  13. 2025Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answeringcited by 2
  14. 2026All Changes May Have Invariant Principles: Improving Ever-Shifting Harmful Meme Detection via Design Concept Reproductioncited by 1
  15. 2025Fall into a Pit, Gain in a Wit: Cognitive-Guided Harmful Meme Detection via Misjudgment Risk Pattern Retrievalcited by 0
  16. 2026Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memescited by 0
  17. 2026SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detectioncited by 0