mlsanityData to 7 Oct 2026

Caption Enriched Samples for Improving Hateful Memes Detection

arXiv 2109.10649 ↗First posted 22 Sep 2021cs.CV cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

The recently introduced hateful meme challenge demonstrates the difficulty of determining whether a meme is hateful or not. Specifically, both unimodal language models and multimodal vision-language models cannot reach the human level of performance. Motivated by the need to model the contrast between the image content and the overlayed text, we suggest applying an off-the-shelf image captioning tool in order to capture the first. We demonstrate that the incorporation of such automatic captions during fine-tuning improves the results for various unimodal and multimodal models. Moreover, in the unimodal case, continuing the pre-training of language models on augmented and original caption pairs, is highly beneficial to the classification accuracy.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 14 of its 15 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

14 of 15 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  11. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  12. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
2 more
  1. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  2. 2020Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memescited by 50

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 4; 2025: 3.

14 papers.

Sort:

By area:

  1. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 34
  2. 2022What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memescited by 18
  3. 2023Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?cited by 14
  4. 2024XMeCap: Meme Caption Generation with Sub-Image Adaptabilitycited by 14
  5. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  6. 2024Towards Comprehensive Detection of Chinese Harmful Memescited by 12
  7. 2025Cultural Bias Matters: A Cross-Cultural Benchmark Dataset and Sentiment-Enriched Model for Understanding Multimodal Metaphorscited by 10
  8. 2023MemeFier: Dual-stage Modality Fusion for Image Meme Classificationcited by 6
  9. 2023MemeGraphs: Linking Memes to Knowledge Graphscited by 6
  10. 2022MetricBERT: Text Representation Learning via Self-Supervised Triplet Trainingcited by 3
  11. 2025Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answeringcited by 2
  12. 2025On VLMs for Diverse Tasks in Multimodal Meme Classificationcited by 2
2 more
  1. 2023Unimodal Intermediate Training for Multimodal Meme Sentiment Classificationcited by 1
  2. 2024Prompt-enhanced Network for Hateful Meme Classificationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection
  2. 2023Mapping Memes to Words for Multimodal Hateful Meme Classification
  3. 2023A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge
  4. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models
  5. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions
  6. 2026GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection
  7. 2024HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes
  8. 2025Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated Knowledge