mlsanityData to 14 Oct 2026

Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study

arXiv 2505.15389 ↗First posted 21 May 2025cs.CL cs.CR cs.CVcited by 12

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Rapid deployment of vision-language models (VLMs) magnifies safety risks, yet most evaluations rely on artificial images. This study asks: How safe are current VLMs when confronted with meme images that ordinary users share? To investigate this question, we introduce MemeSafetyBench, a 50,430-instance benchmark pairing real meme images with both harmful and benign instructions. Using a comprehensive safety taxonomy and LLM-based instruction generation, we assess multiple VLMs across single and multi-turn interactions. We investigate how real-world memes influence harmful outputs, the mitigating effects of conversational context, and the relationship between model scale and safety metrics. Our findings demonstrate that VLMs are more vulnerable to meme-based harmful prompts than to synthetic or typographic images. Memes significantly increase harmful responses and decrease refusals compared to text-only inputs. Though multi-turn interactions provide partial mitigation, elevated vulnerability persists. These results highlight the need for ecologically valid evaluations and stronger safety mechanisms. MemeSafetyBench is publicly available at https://github.com/oneonlee/Meme-Safety-Bench.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 63 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

31 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs17 May
  2. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations14 Feb
  3. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Prompting25 Dec
  4. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts21 Jul
  5. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 26

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  6. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  7. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  8. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  9. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
  10. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 86
  11. 2020SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!cited by 78
  12. 2021Detecting Harmful Memes and Their Targetscited by 78
14 more
  1. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 71
  2. 2022Detecting and Understanding Harmful Memes: A Surveycited by 58
  3. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  4. 2021SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Imagescited by 50
  5. 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
  6. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
  7. 2024MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classificationcited by 30
  8. 2023Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Modelscited by 26
  9. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  10. 2023Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learningcited by 21
  11. 2025MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attackscited by 10
  12. 2025Detecting and Filtering Unsafe Training Data via Data Attribution with Denoised Representationcited by 9
  13. 2025sudo rm -rf agentic_securitycited by 8
  14. 2025Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attackingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 2025Jailbreaking on Text-to-Video Models via Scene Splitting Strategycited by 12
  2. 2026MemeLens: Multilingual Multitask VLMs for Memescited by 6
  3. 2026COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMscited by 2
  4. 2026Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classificationcited by 1
  5. 2026Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMscited by 1
  6. 2026ViMU: Benchmarking Video Metaphorical Understandingcited by 1
  7. 2026SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safetycited by 0
  8. 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challengescited by 0
  9. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  10. 2026MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Modelscited by 0
  11. 2026Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Modelscited by 0
  12. 2026The Uncontrolled Variable: Vision-Language Model Refusal Responds to Image Presence in Ways Risk Cannot Explaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Models
  2. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models
  3. 2026When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Life
  4. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
  5. 2025AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness
  6. 2026From Native Memes to Global Moderation: Cross-Cultural Evaluation of Vision-Language Models for Hateful Meme Detection
  7. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models
  8. 2024Safety of Multimodal Large Language Models on Images and Texts