Multimodal Safety Evaluation Should Measure Controllability Beyond Classification
Not on the map. Why?
Abstract (arXiv)
VLM safety is commonly evaluated through input- and output-level classification. Such classification is necessary, but it does not reveal whether a safety state is accessible or controllable inside the model. We argue that multimodal safety evaluation should therefore report a controllability profile alongside behavioral classification, separating representation-level detectability, cross-modal specificity, intervention sensitivity, and benign-preserving selectivity. Using implicit toxicity as a stress case, we instantiate this profile on LlavaGuard and Qwen3.5 with sparse feature decompositions. LlavaGuard admits localized handles with a narrow benign-preserving intervention range and modest downstream safety gains, whereas Qwen3.5 supports strong representation-level readout but no comparable selective-control regime under the tested operators. These results show that internal readout and controllability can diverge. Future multimodal safety benchmarks should therefore report not only behavioral safety metrics, but also whether safety-relevant internal signals can be intervention-tested and controlled within a validated operating range.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 59 references that are in the map. First posted on arXiv 5 Oct 2026.
Builds on
13 of 59 references.
References · 13
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
- 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
- 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judgescited by 24
- 2025When Language Overrules: Revealing Text Dominance in Multimodal Large Language Modelscited by 18
- 2025JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Modelscited by 10
- 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Modelcited by 9
- 2025ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMscited by 7
- 2025DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Modelscited by 6
- 2025Less is More: Improving LLM Reasoning with Minimal Test-Time Interventioncited by 5
- 2025CEE: An Inference-Time Jailbreak Defense for Embodied Intelligence via Subspace Concept Rotationcited by 4
- 2026When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Modelscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Do Model Internals Help? Exploring the Role of Representation Engineering in LLM Safety
- 2026MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
- 2026Don't Judge an LLM Only by Its Activations: Discovering Suppressed Safety Features via Counterfactual Activation Potential
- 2026From Latent Space to Jacobian Space: Measuring, Evading, and Training Against Safety-Content Accessibility
- 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback
- 2026LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories
- 2026Reading, Not Manipulating: Leveraging Router Logits for Multimodal Safety in MoE Vision-Language Models
- 2026ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models