mlsanityData to 14 Oct 2026

Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model

arXiv 2311.06737 ↗First posted 12 Nov 2023cs.CL cs.AIcited by 12

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recently, large language models (LLMs) have taken the spotlight in natural language processing. Further, integrating LLMs with vision enables the users to explore more emergent abilities in multimodality. Visual language models (VLMs), such as LLaVA, Flamingo, or GPT-4, have demonstrated impressive performance on various visio-linguistic tasks. Consequently, there are enormous applications of large models that could be potentially used on social media platforms. Despite that, there is a lack of related work on detecting or correcting hateful memes with VLMs. In this work, we study the ability of VLMs on hateful meme detection and hateful meme correction tasks with zero-shot prompting. From our empirical experiments, we show the effectiveness of the pretrained LLaVA model and discuss its strengths and weaknesses in these tasks.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 36 references that are in the map. First posted on arXiv 12 Nov 2023.

Builds on

28 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep

Other references · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
15 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  5. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  6. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  7. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 407
  8. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  9. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  10. 2020Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memescited by 51
  11. 2020Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challengecited by 51
  12. 2020A Multimodal Framework for the Detection of Hateful Memescited by 49
  13. 2020Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solutioncited by 41
  14. 2020Detecting Hateful Memes Using a Multimodal Deep Ensemblecited by 30
  15. 2022Why Is It Hate Speech? Masked Rationale Prediction for Explainable Hate Speech Detectioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 2; 2026: 1.

12 papers.

  1. 2024Safety of Multimodal Large Language Models on Images and Textscited by 75
  2. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  3. 2024EmoLLM: Multimodal Emotional Understanding Meets Large Language Modelscited by 31
  4. 2024Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Modelscited by 26
  5. 2024What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot Detectioncited by 25
  6. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  7. 2024Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretationscited by 10
  8. 2024LLMs are One-Shot URL Classifiers and Explainerscited by 6
  9. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memescited by 4
  10. 2025MV-Debate: Multi-view Agent Debate with Dynamic Reflection Gating for Multimodal Harmful Content Detection in Social Mediacited by 4
  11. 2025SocialDF: Benchmark Dataset and Detection Model for Mitigating Harmful Deepfake Content on Social Media Platformscited by 2
  12. 2026Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarkingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models
  2. 2026Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memes
  3. 2025Labels or Input? Rethinking Augmentation in Multimodal Hate Detection
  4. 2023Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection
  5. 2025On VLMs for Diverse Tasks in Multimodal Meme Classification
  6. 2023A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge
  7. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions
  8. 2026FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detection