mlsanityData to 7 Oct 2026

MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets

arXiv 2109.05184 ↗First posted 11 Sep 2021cs.MM cs.CLcited by 85

Not on the map. Why?

Abstract (arXiv)

Internet memes have become powerful means to transmit political, psychological, and socio-cultural ideas. Although memes are typically humorous, recent days have witnessed an escalation of harmful memes used for trolling, cyberbullying, and abuse. Detecting such memes is challenging as they can be highly satirical and cryptic. Moreover, while previous work has focused on specific aspects of memes such as hate speech and propaganda, there has been little work on harm in general. Here, we aim to bridge this gap. We focus on two tasks: (i)detecting harmful memes, and (ii)identifying the social entities they target. We further extend a recently released HarMeme dataset, which covered COVID-19, with additional memes and a new topic: US politics. To solve these tasks, we propose MOMENTA (MultimOdal framework for detecting harmful MemEs aNd Their tArgets), a novel multimodal deep neural network that uses global and local perspectives to detect harmful memes. MOMENTA systematically analyzes the local and the global perspective of the input meme (in both modalities) and relates it to the background context. MOMENTA is interpretable and generalizable, and our experiments show that it outperforms several strong rivaling approaches.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 46 of its 75 references that are in the map. First posted on arXiv 11 Sep 2021.

Builds on

46 of 75 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2016Densely Connected Convolutional Networkscited by 6,691
  12. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
34 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  9. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  10. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  11. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  12. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  13. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  14. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  15. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  16. 2020Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT modelcited by 87
  17. 2020SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!cited by 77
  18. 2020Demoting Racial Bias in Hate Speech Detectioncited by 62
  19. 2020Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challengecited by 51
  20. 2020Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memescited by 50
  21. 2021SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Imagescited by 49
  22. 2020A Multimodal Framework for the Detection of Hateful Memescited by 48
  23. 2021Detecting Propaganda Techniques in Memescited by 43
  24. 2020What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Contextcited by 42
  25. 2015What value do explicit high level concepts have in vision to language problems?cited by 31
  26. 2020Detecting Hateful Memes Using a Multimodal Deep Ensemblecited by 30
  27. 2019Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderationcited by 29
  28. 2020Multimodal Learning for Hateful Memes Detectioncited by 29
  29. 2021"Subverting the Jewtocracy": Online Antisemitism Detection Using Multimodal Deep Learningcited by 21
  30. 2018A Deep Learning Approach for Multimodal Deception Detectioncited by 14
  31. 2019Predicting the Leading Political Ideology of YouTube Channels Using Acoustic, Textual, and Metadata Informationcited by 9
  32. 2020Detecting Medical Misinformation on Social Media Using Multimodal Deep Learningcited by 9
  33. 2021Exercise? I thought you said 'Extra Fries': Leveraging Sentence Demarcations and Multi-hop Attention for Meme Affect Analysiscited by 9
  34. 2019Detecting Deception in Political Debates Using Acoustic and Textual Featurescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 12; 2023: 17; 2024: 18; 2025: 18; 2026: 19.

85 papers.

Sort:

By area:

  1. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  2. 2022Handling Bias in Toxic Speech Detection: A Surveycited by 60
  3. 2022Where in the World is this Image? Transformer-based Geo-localization in the Wildcited by 58
  4. 2022Detecting and Understanding Harmful Memes: A Surveycited by 58
  5. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
  6. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 50
  7. 2023Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detectioncited by 45
  8. 2023Decoding the Underlying Meaning of Multimodal Hateful Memescited by 38
  9. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  10. 2021Multimodal Learning using Optimal Transport for Sarcasm and Humor Detectioncited by 36
  11. 2024MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classificationcited by 29
  12. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
38 more
  1. 2024Towards Low-Resource Harmful Meme Detection with LMM Agentscited by 23
  2. 2022DISARM: Detecting the Victims Targeted by Harmful Memescited by 20
  3. 2023Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learningcited by 20
  4. 2024MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Interventioncited by 20
  5. 2022What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memescited by 18
  6. 2023BanglaAbuseMeme: A Dataset for Bengali Abusive Meme Classificationcited by 18
  7. 2024Modularized Networks for Few-shot Hateful Meme Detectioncited by 15
  8. 2023Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?cited by 14
  9. 2025MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detectioncited by 14
  10. 2024Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanationscited by 13
  11. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  12. 2024Multimodal Pragmatic Jailbreak on Text-to-image Modelscited by 13
  13. 2023Memotion 3: Dataset on Sentiment and Emotion Analysis of Codemixed Hindi-English Memescited by 12
  14. 2024MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencingcited by 12
  15. 2024Towards Comprehensive Detection of Chinese Harmful Memescited by 12
  16. 2025Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Studycited by 12
  17. 2023MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualizationcited by 11
  18. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detectioncited by 11
  19. 2023A Template Is All You Memecited by 9
  20. 2023Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devicescited by 8
  21. 2024Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detectioncited by 8
  22. 2025MemeBLIP2: A novel lightweight multimodal system to detect harmful memescited by 8
  23. 2023Exploring news intent and its application: A theory-driven approachcited by 7
  24. 2024Deciphering Hate: Identifying Hateful Memes and Their Targetscited by 7
  25. 2024Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memescited by 7
  26. 2023MemeGraphs: Linking Memes to Knowledge Graphscited by 6
  27. 2025Figurative-cum-Commonsense Knowledge Infusion for Multimodal Mental Health Meme Classificationcited by 6
  28. 2025ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detectioncited by 6
  29. 2023PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Modelscited by 5
  30. 2022Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Resultscited by 4
  31. 2024Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detectioncited by 4
  32. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memescited by 4
  33. 2025MemeReaCon: Probing Contextual Meme Understanding in Large Vision-Language Modelscited by 4
  34. 2025A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehensioncited by 4
  35. 2025From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignmentcited by 3
  36. 2022Nipping in the Bud: Detection, Diffusion and Mitigation of Hate Speech on Social Mediacited by 2
  37. 2022hate-alert@DravidianLangTech-ACL2022: Ensembling Multi-Modalities for Tamil TrollMeme Classificationcited by 2
  38. 2022Detecting the Role of an Entity in Harmful Memes: Techniques and Their Limitationscited by 2

50 most cited of 85 All 85 (JSON)