mlsanityData to 7 Oct 2026

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

arXiv 2005.04790 ↗First posted 10 May 2020cs.AI cs.CL cs.CVcited by 485

Not on the map. Why?

Cited by 113 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples ("benign confounders") are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans (64.73% vs. 84.7% accuracy), illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.

Key facts. Cited by 485 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 45; 2022: 57; 2023: 79; 2024: 102; 2025: 108; 2026: 94.

485 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  6. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  12. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
38 more
  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  2. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  3. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  4. 2022Multimodal Learning with Transformers: A Surveycited by 399
  5. 2024What matters when building vision-language models?cited by 321
  6. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  8. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  9. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  10. 2022A Self-Supervised Descriptor for Image Copy Detectioncited by 175
  11. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  12. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  13. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  14. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  15. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  16. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  17. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  18. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  19. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  20. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  21. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  22. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  23. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  24. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  25. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  26. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  27. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  28. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 102
  29. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  30. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  31. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  32. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 85
  33. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  34. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  35. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  36. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  37. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  38. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77

50 most cited of 485 All 485 (JSON)