mlsanityData to 7 Oct 2026

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge

arXiv 2012.12975 ↗First posted 23 Dec 2020cs.AI cs.CL cs.CVcited by 51

Not on the map. Why?

Abstract (arXiv)

Memes on the Internet are often harmless and sometimes amusing. However, by using certain types of images, text, or combinations of both, the seemingly harmless meme becomes a multimodal type of hate speech -- a hateful meme. The Hateful Memes Challenge is a first-of-its-kind competition which focuses on detecting hate speech in multimodal memes and it proposes a new data set containing 10,000+ new examples of multimodal content. We utilize VisualBERT -- which meant to be the BERT of vision and language -- that was trained multimodally on images and captions and apply Ensemble Learning. Our approach achieves 0.811 AUROC with an accuracy of 0.765 on the challenge test set and placed third out of 3,173 participants in the Hateful Memes Challenge.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 9; 2023: 12; 2024: 6; 2025: 10; 2026: 7.

51 papers.

Sort:

By area:

  1. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 85
  2. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  3. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
  4. 2021Disentangling Hate in Online Memescited by 46
  5. 2023Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detectioncited by 45
  6. 2022On Explaining Multimodal Hateful Meme Detection Modelscited by 29
  7. 2024Towards Low-Resource Harmful Meme Detection with LMM Agentscited by 23
  8. 2023Mapping Memes to Words for Multimodal Hateful Meme Classificationcited by 19
  9. 2022Emotion Based Hate Speech Detection using Multimodal Learningcited by 18
  10. 2022What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memescited by 18
  11. 2023BanglaAbuseMeme: A Dataset for Bengali Abusive Meme Classificationcited by 18
  12. 2024Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Modelscited by 17
38 more
  1. 2024Modularized Networks for Few-shot Hateful Meme Detectioncited by 15
  2. 2023Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Modelscited by 14
  3. 2025MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detectioncited by 14
  4. 2021AOMD: An Analogy-aware Approach to Offensive Meme Detection on Social Mediacited by 13
  5. 2022DisinfoMeme: A Multimodal Dataset for Detecting Meme Intentionally Spreading Out Disinformationcited by 13
  6. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  7. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisionscited by 13
  8. 2023Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Modelcited by 12
  9. 2023MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualizationcited by 11
  10. 2025MemeIntel: Explainable Detection of Propagandistic and Hateful Memescited by 8
  11. 2023HateProof: Are Hateful Meme Detection Systems really Robust?cited by 7
  12. 2023MemeFier: Dual-stage Modality Fusion for Image Meme Classificationcited by 6
  13. 2025MemeReaCon: Probing Contextual Meme Understanding in Large Vision-Language Modelscited by 4
  14. 2021An Interpretable Approach to Hateful Meme Detectioncited by 3
  15. 2021Towards A Multi-agent System for Online Hate Speech Detectioncited by 2
  16. 2022Captcha Attack: Turning Captchas Against Humanitycited by 2
  17. 2022DeepGraviLens: a Multi-Modal Architecture for Classifying Gravitational Lensing Datacited by 2
  18. 2023TOT: Topology-Aware Optimal Transport For Multimodal Hate Detectioncited by 2
  19. 2025An Ensemble Model with Attention Based Mechanism for Image Captioningcited by 2
  20. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Modelscited by 2
  21. 2025Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computingcited by 2
  22. 2026Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learningcited by 2
  23. 2021Hateful Memes Challenge: An Enhanced Multimodal Frameworkcited by 1
  24. 2021Visual Persuasion in COVID-19 Social Media Content: A Multi-Modal Characterizationcited by 1
  25. 2022Feels Bad Man: Dissecting Automated Hateful Meme Detection Through the Lens of Facebook's Challengecited by 1
  26. 2022The Hateful Memes Challenge Next Movecited by 1
  27. 2023A Review of Vision-Language Models and their Performance on the Hateful Memes Challengecited by 1
  28. 2023Lexical Squad@Multimodal Hate Speech Event Detection 2023: Multimodal Hate Speech Detection using Fused Ensemble Approachcited by 1
  29. 2024COT: A Generative Approach for Hate Speech Counter-Narratives via Contrastive Optimal Transportcited by 1
  30. 2024Prompt-enhanced Network for Hateful Meme Classificationcited by 1
  31. 2025DeHate: A Stable Diffusion-based Multimodal Approach to Mitigate Hate Speech in Imagescited by 1
  32. 2025Ideology-Based LLMs for Content Moderationcited by 1
  33. 2025Unpacking Hateful Memes: Presupposed Context and False Claimscited by 0
  34. 2026Unveiling Covert Toxicity in Multimodal Data via Toxicity Association Graphs: A Graph-Based Metric and Interpretable Detection Frameworkcited by 0
  35. 2026MemeScouts@LT-EDI 2026: Asking the Right Questions -- Prompted Weak Supervision for Meme Hate Speech Detectioncited by 0
  36. 2026PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Frameworkcited by 0
  37. 2026RMS@CC-MMD 2026: Multimodal Misogyny Detection via Geometric Interaction and Multi-View Consensuscited by 0
  38. 2026Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memescited by 0

50 most cited of 51 All 51 (JSON)