mlsanityData to 7 Oct 2026

SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models

arXiv 2508.06142 ↗First posted 8 Aug 2025cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

In the rapidly evolving landscape of Multimodal Large Language Models (MLLMs), the safety concerns of their outputs have earned significant attention. Although numerous datasets have been proposed, they may become outdated with MLLM advancements and are susceptible to data contamination issues. To address these problems, we propose SDEval, the textitfirst safety dynamic evaluation framework to controllably adjust the distribution and complexity of safety benchmarks. Specifically, SDEval mainly adopts three dynamic strategies: text, image, and text-image dynamics to generate new samples from original benchmarks. We first explore the individual effects of text and image dynamics on model safety. Then, we find that injecting text dynamics into images can further impact safety, and conversely, injecting image dynamics into text also leads to safety risks. SDEval is general enough to be applied to various existing safety and even capability benchmarks. Experiments across safety benchmarks, MLLMGuard and VLSBench, and capability benchmarks, MMBench and MMVet, show that SDEval significantly influences safety evaluation, mitigates data contamination, and exposes safety limitations of MLLMs. Code is available at https://github.com/hq-King/SDEval

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 49 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

27 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2024Yi: Open Foundation Models by 01.AIcited by 706
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  8. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  9. 2023An Overview of Catastrophic AI Riskscited by 214
  10. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 199
  11. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  12. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
14 more
  1. 2025Seedream 3.0 Technical Reportcited by 146
  2. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  3. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  4. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  5. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  6. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  7. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrappingcited by 22
  8. 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoningcited by 19
  9. 2022A Coding Framework and Benchmark towards Low-Bitrate Video Understandingcited by 17
  10. 2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 14
  11. 2024SMC++: Masked Learning of Unsupervised Video Semantic Compressioncited by 11
  12. 2025Towards All-in-One Medical Image Re-Identificationcited by 6
  13. 2024Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignmentcited by 5
  14. 2025Semantics versus Identity: A Divide-and-Conquer Approach towards Adjustable Medical Image De-Identificationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026UniHM: Unified Dexterous Hand Manipulation with Vision Language Modelcited by 3
  2. 2026Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMscited by 1
  3. 2025HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal Large Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models
  2. 2024Safety of Multimodal Large Language Models on Images and Texts
  3. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models
  4. 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
  5. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safety
  6. 2025Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
  7. 2025Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?
  8. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues