mlsanityData to 14 Oct 2026

VSCBench: Bridging the Gap in Vision-Language Model Safety Calibration

arXiv 2505.20362 ↗First posted 26 May 2025cs.IR cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

The rapid advancement of vision-language models (VLMs) has brought a lot of attention to their safety alignment. However, existing methods have primarily focused on model undersafety, where the model responds to hazardous queries, while neglecting oversafety, where the model refuses to answer safe queries. In this paper, we introduce the concept of textitsafety calibration, which systematically addresses both undersafety and oversafety. Specifically, we present VSCBench, a novel dataset of 3,600 image-text pairs that are visually or textually similar but differ in terms of safety, which is designed to evaluate safety calibration across image-centric and text-centric scenarios. Based on our benchmark, we evaluate safety calibration across eleven widely used VLMs. Our extensive experiments revealed major issues with both undersafety and oversafety. We further investigated four approaches to improve the model's safety calibration. We found that even though some methods effectively calibrated the models' safety problems, these methods also lead to the degradation of models' utility. This trade-off underscores the urgent need for advanced calibration methods, and our benchmark provides a valuable tool for evaluating future approaches. Our code and data are available at https://github.com/jiahuigeng/VSCBench.git.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 34 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

17 of 34 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Update24 Jan
  2. 2024MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?22 Jun
  3. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  4. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
  5. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  6. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep

Other references · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  10. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  11. 2021Detecting Harmful Content On Online Platforms: What Platforms Need Vs. Where Research Efforts Gocited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalitiescited by 7
  2. 2025DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Modelscited by 6
  3. 2026SEA-Guard: Culturally Grounded Multilingual Safeguard for Southeast Asiacited by 2
  4. 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequencescited by 0
  5. 2026PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrailscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model
  2. 2025Safe Vision-Language Models via Unsafe Weights Manipulation
  3. 2024Safety Alignment for Vision Language Models
  4. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model
  5. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models
  6. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safety
  7. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
  8. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models