mlsanityData to 14 Oct 2026

SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models

arXiv 2410.18927 ↗First posted 24 Oct 2024cs.CRcited by 69

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) are showing strong safety concerns (e.g., generating harmful outputs for users), which motivates the development of safety evaluation benchmarks. However, we observe that existing safety benchmarks for MLLMs show limitations in query quality and evaluation reliability limiting the detection of model safety implications as MLLMs continue to evolve. In this paper, we propose toolns, a comprehensive framework designed for conducting safety evaluations of MLLMs. Our framework consists of a comprehensive harmful query dataset and an automated evaluation protocol that aims to address the above limitations, respectively. We first design an automatic safety dataset generation pipeline, where we employ a set of LLM judges to recognize and categorize the risk scenarios that are most harmful and diverse for MLLMs; based on the taxonomy, we further ask these judges to generate high-quality harmful queries accordingly resulting in 23 risk scenarios with 2,300 multi-modal harmful query pairs. During safety evaluation, we draw inspiration from the jury system in judicial proceedings and pioneer the jury deliberation evaluation protocol that adopts collaborative LLMs to evaluate whether target models exhibit specific harmful behaviors, providing a reliable and unbiased assessment of content security risks. In addition, our benchmark can also be extended to the audio modality showing high scalability and potential. Based on our framework, we conducted large-scale experiments on 15 widely-used open-source MLLMs and 6 commercial MLLMs (e.g., GPT-4o, Gemini), where we revealed widespread safety issues in existing MLLMs and instantiated several insights on MLLM safety performance such as image quality and parameter size.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 69 of its 113 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

69 of 113 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Does Refusal Training in LLMs Generalize to the Past Tense?16 Jul
  2. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queries23 Feb

Other references · 67

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
38 more
  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  10. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  11. 2024Yi: Open Foundation Models by 01.AIcited by 719
  12. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  13. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  14. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  15. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  16. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  17. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  18. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  19. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  20. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  21. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  22. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  23. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  24. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  25. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  26. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 251
  27. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  28. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  29. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  30. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  31. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  32. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183
  33. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  34. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  35. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 122
  36. 2023BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learningcited by 115
  37. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 109
  38. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 41; 2026: 26.

69 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 75
  3. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 67
  4. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 64
  5. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  6. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  7. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 44
  8. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 38
  9. 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselvescited by 37
  10. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 36
  11. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  12. 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
38 more
  1. 2025Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findingscited by 27
  2. 2025o3-mini vs DeepSeek-R1: Which One is Safer?cited by 24
  3. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  4. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 18
  5. 2025RoboSafe: Safeguarding Embodied Agents via Executable Safety Logiccited by 14
  6. 2025Early External Safety Testing of OpenAI's o3-mini: Insights from the Pre-Deployment Evaluationcited by 13
  7. 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Modelscited by 13
  8. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 12
  9. 2025MORALISE: A Structured Benchmark for Moral Alignment in Visual Language Modelscited by 10
  10. 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Modelscited by 10
  11. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editingcited by 10
  12. 2025Multimodal Cultural Safety: Evaluation Framework and Alignment Strategiescited by 8
  13. 2025Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Cipherscited by 8
  14. 2025ASTRAL: Automated Safety Testing of Large Language Modelscited by 7
  15. 2025Universal Adversarial Attack on Aligned Multimodal LLMscited by 6
  16. 2025POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Modelscited by 6
  17. 2025Quantifying CBRN Risk in Frontier Modelscited by 6
  18. 2025Failures to Surface Harmful Contents in Video Large Language Modelscited by 5
  19. 2025SPARK: Jailbreaking T2V Models by Synergistically Prompting Auditory and Recontextualized Knowledgecited by 4
  20. 2025GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Supervisioncited by 4
  21. 2026Retrieval Collapses When AI Pollutes the Webcited by 4
  22. 2026BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environmentscited by 4
  23. 2025PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreakingcited by 3
  24. 2025From Evaluation to Defense: Advancing Safety in Video Large Language Modelscited by 2
  25. 2025PRJ: Perception-Retrieval-Judgement for Generated Imagescited by 2
  26. 2025RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulationcited by 2
  27. 2025Falcon: A Cross-Modal Evaluation Dataset for Comprehensive Safety Perceptioncited by 2
  28. 2025Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytellingcited by 2
  29. 2025SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Drivingcited by 2
  30. 2026Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Modelscited by 2
  31. 2026Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teamingcited by 2
  32. 2026Safe responses matter: Output-aware safety guardrail mitigate over-refusal in MLLMscited by 2
  33. 2025Benchmarking the Trustworthiness in Multimodal LLMs for Video Understandingcited by 1
  34. 2025LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Modelscited by 1
  35. 2026Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMscited by 1
  36. 2026Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspectivecited by 1
  37. 2026SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreakscited by 1
  38. 2025Data Security in Large Language Models: Risks, Defense, and Directionscited by 0

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SafeLawBench: Towards Safe Alignment of Large Language Models
  2. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models
  3. 2024S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models
  4. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models
  5. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety
  6. 2025SafeMT: Multi-turn Safety for Multimodal Language Models
  7. 2025SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models
  8. 2023All Languages Matter: On the Multilingual Safety of Large Language Models