mlsanityData to 14 Oct 2026

GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abuse

arXiv 2401.01523 ↗First posted 3 Jan 2024cs.CL cs.AIcited by 52

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The exponential growth of social media has profoundly transformed how information is created, disseminated, and absorbed, exceeding any precedent in the digital age. Regrettably, this explosion has also spawned a significant increase in the online abuse of memes. Evaluating the negative impact of memes is notably challenging, owing to their often subtle and implicit meanings, which are not directly conveyed through the overt text and image. In light of this, large multimodal models (LMMs) have emerged as a focal point of interest due to their remarkable capabilities in handling diverse multimodal tasks. In response to this development, our paper aims to thoroughly examine the capacity of various LMMs (e.g., GPT-4o) to discern and respond to the nuanced aspects of social abuse manifested in memes. We introduce the comprehensive meme benchmark, GOAT-Bench, comprising over 6K varied memes encapsulating themes such as implicit hate speech, sexism, and cyberbullying, etc. Utilizing GOAT-Bench, we delve into the ability of LMMs to accurately assess hatefulness, misogyny, offensiveness, sarcasm, and harmful content. Our extensive experiments across a range of LMMs reveal that current models still exhibit a deficiency in safety awareness, showing insensitivity to various forms of implicit abuse. We posit that this shortfall represents a critical impediment to the realization of safe artificial intelligence. The GOAT-Bench and accompanying resources are publicly accessible at https://goatlmm.github.io/, contributing to ongoing research in this vital field.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 51 of its 87 references that are in the map. First posted on arXiv 3 Jan 2024.

Builds on

51 of 87 references.

References · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  11. 2023LIMA: Less Is More for Alignmentcited by 1,100
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  14. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  15. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  16. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  17. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  18. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  19. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  20. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  21. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  22. 2018nocaps: novel object captioning at scalecited by 444
  23. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  24. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  25. 2022UL2: Unifying Language Learning Paradigmscited by 295
  26. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  27. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  28. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  29. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  30. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  31. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 86
  32. 2021Detecting Harmful Memes and Their Targetscited by 78
  33. 2022Detecting and Understanding Harmful Memes: A Surveycited by 58
  34. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  35. 2022Detect Rumors in Microblog Posts for Low-Resource Domains via Adversarial Contrastive Learningcited by 41
  36. 2022Zero-Shot Rumor Detection with Propagation Structure via Prompt Learningcited by 32
  37. 2021Rumor Detection on Twitter with Claim-Guided Hierarchical Graph Attention Networkscited by 29
  38. 2023Memotion 3: Dataset on Sentiment and Emotion Analysis of Codemixed Hindi-English Memescited by 12

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 20; 2025: 15; 2026: 17.

52 papers.

Sort:

By area:

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  2. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  3. 2024Safety of Multimodal Large Language Models on Images and Textscited by 75
  4. 2024Red Teaming Visual Language Modelscited by 65
  5. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  6. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  7. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  8. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  9. 2024Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Modelcited by 26
  10. 2024Towards Low-Resource Harmful Meme Detection with LMM Agentscited by 24
  11. 2024A Survey of Multimodal Sarcasm Detectioncited by 19
  12. 2024Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Modelscited by 18
38 more
  1. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Modelscited by 15
  2. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisionscited by 14
  3. 2024Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicitiescited by 13
  4. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Modelscited by 12
  5. 2024Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretationscited by 10
  6. 2024CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Modelscited by 8
  7. 2025ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMscited by 7
  8. 2025Can Large Vision-Language Models Understand Multimodal Sarcasm?cited by 5
  9. 2024ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challengescited by 4
  10. 2025MemeReaCon: Probing Contextual Meme Understanding in Large Vision-Language Modelscited by 4
  11. 2025Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalitiescited by 4
  12. 2025Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigationcited by 4
  13. 2026PEST: Parameter Efficient Steering of Blackbox VLMs via Agentic Few-shot Alignment for Hateful Meme Moderationcited by 4
  14. 2025Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detectioncited by 3
  15. 2026M-QUEST -- Meme Question-Understanding Evaluation on Semantics and Toxicitycited by 3
  16. 2024Can Large Language Models Automatically Jailbreak GPT-4V?cited by 2
  17. 2024HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memescited by 2
  18. 2025SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Servicescited by 2
  19. 2025Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarioscited by 2
  20. 2025Evaluating Multimodal Large Language Models on Spoken Sarcasm Understandingcited by 2
  21. 2025MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critiquecited by 2
  22. 2025MDIT-Bench: Evaluating the Dual-Implicit Toxicity in Large Multimodal Modelscited by 1
  23. 2026All Changes May Have Invariant Principles: Improving Ever-Shifting Harmful Meme Detection via Design Concept Reproductioncited by 1
  24. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMscited by 1
  25. 2026MER-Bench: A Comprehensive Benchmark for Multimodal Meme Reappraisalcited by 1
  26. 2026The Overlooked Repetitive Lengthening Form in Sentiment Analysiscited by 1
  27. 2026ViMU: Benchmarking Video Metaphorical Understandingcited by 1
  28. 2026Small Cues, Big Consequences: Learning Pivotal Cues for Multimodal Meme Classificationcited by 1
  29. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0
  30. 2025STEMTOX: From Collaborative Tags to Fine-Grained Toxic Meme Detection via Entropy-Guided Multi-Task Learningcited by 0
  31. 2025MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Modelscited by 0
  32. 2026Ethical Risks in Deploying Large Language Models: An Evaluation of Medical Ethics Jailbreakingcited by 0
  33. 2026Unveiling Covert Toxicity in Multimodal Data via Toxicity Association Graphs: A Graph-Based Metric and Interpretable Detection Frameworkcited by 0
  34. 2026Beyond Hate: Differentiating Uncivil and Intolerant Speech in Multimodal Content Moderationcited by 0
  35. 2026FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detectioncited by 0
  36. 2026I Know What You Meme, Even If it Emerged Today: Understanding Evolving Memes through Open-World Knowledge Acquisitioncited by 0
  37. 2026SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Contextcited by 0
  38. 2026Towards High-Level Semantic Intelligencecited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness
  2. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memes
  3. 2025From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignment
  4. 2023A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge
  5. 2026They Said Memes Were Harmless-We Found the Ones That Hurt: Decoding Jokes, Symbols, and Cultural References
  6. 2024MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Intervention
  7. 2025A survey of textual cyber abuse detection using cutting-edge language models and large language models
  8. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models