mlsanityData to 7 Oct 2026

MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models

arXiv 2311.17600 ↗First posted 29 Nov 2023cs.CVcited by 255

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

The security concerns surrounding Large Language Models (LLMs) have been extensively explored, yet the safety of Multimodal Large Language Models (MLLMs) remains understudied. In this paper, we observe that Multimodal Large Language Models (MLLMs) can be easily compromised by query-relevant images, as if the text query itself were malicious. To address this, we introduce MM-SafetyBench, a comprehensive framework designed for conducting safety-critical evaluations of MLLMs against such image-based manipulations. We have compiled a dataset comprising 13 scenarios, resulting in a total of 5,040 text-image pairs. Our analysis across 12 state-of-the-art models reveals that MLLMs are susceptible to breaches instigated by our approach, even when the equipped LLMs have been safety-aligned. In response, we propose a straightforward yet effective prompting strategy to enhance the resilience of MLLMs against these types of attacks. Our work underscores the need for a concerted effort to strengthen and enhance the safety measures of open-source MLLMs against potential malicious exploits. The resource is available at https://github.com/isXinLiu/MM-SafetyBench

Key facts. Cited by 255 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 50 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

43 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  2. 2023Can Language Models be Instructed to Protect Personal Information?3 Oct

Other references · 41

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
29 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  9. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  10. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  11. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  13. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  14. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  15. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  16. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  17. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  18. 2023On the Exploitability of Instruction Tuningcited by 110
  19. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  20. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  21. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  22. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Promptscited by 62
  23. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  24. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 54
  25. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  26. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 30
  27. 2023SC-Safety: A Multi-round Open-ended Question Adversarial Safety Benchmark for Large Language Models in Chinesecited by 17
  28. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Modelscited by 10
  29. 2023Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 109; 2026: 98.

255 papers.

Sort:

By area:

  1. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  2. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  3. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  4. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  5. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  6. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  7. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 82
  8. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  9. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  10. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  11. 2024Safety of Multimodal Large Language Models on Images and Textscited by 70
  12. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 68
38 more
  1. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  2. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  3. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  4. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  5. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  6. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  7. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 51
  8. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  9. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  10. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 44
  11. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  12. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistencycited by 42
  13. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  14. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 40
  15. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  16. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  17. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  18. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  19. 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselvescited by 34
  20. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 34
  21. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  22. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  23. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  24. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 31
  25. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 29
  26. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  27. 2024MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?cited by 28
  28. 2024Safety Alignment for Vision Language Modelscited by 27
  29. 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
  30. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  31. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 27
  32. 2025ShieldGemma 2: Robust and Tractable Image Content Moderationcited by 27
  33. 2024Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Imagecited by 26
  34. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 25
  35. 2025Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findingscited by 24
  36. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  37. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Modelcited by 22
  38. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Modelscited by 21

50 most cited of 255 All 255 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation
  2. 2025SafeMT: Multi-turn Safety for Multimodal Language Models
  3. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
  4. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration
  5. 2025SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models
  6. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues
  7. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
  8. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safety