# Multimodal Safety Evaluation Should Measure Controllability Beyond Classification

arXiv: https://arxiv.org/abs/2610.06452 · HTML: https://mlsanity.com/paper/2610.06452/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 59 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 13 of 13)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges](https://mlsanity.com/paper/2503.04474/) (2025)
- [When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models](https://mlsanity.com/paper/2508.10552/) (2025)
- [JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models](https://mlsanity.com/paper/2505.19610/) (2025)
- [HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model](https://mlsanity.com/paper/2506.04704/) (2025)
- [ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs](https://mlsanity.com/paper/2505.14035/) (2025)
- [DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models](https://mlsanity.com/paper/2510.10846/) (2025)
- [Less is More: Improving LLM Reasoning with Minimal Test-Time Intervention](https://mlsanity.com/paper/2510.13940/) (2025)
- [CEE: An Inference-Time Jailbreak Defense for Embodied Intelligence via Subspace Concept Rotation](https://mlsanity.com/paper/2504.13201/) (2025)
- [When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models](https://mlsanity.com/paper/2605.27851/) (2026)
- [Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs](https://mlsanity.com/paper/2601.15698/) (2026)

## Abstract (arXiv)

VLM safety is commonly evaluated through input- and output-level classification. Such classification is necessary, but it does not reveal whether a safety state is accessible or controllable inside the model. We argue that multimodal safety evaluation should therefore report a controllability profile alongside behavioral classification, separating representation-level detectability, cross-modal specificity, intervention sensitivity, and benign-preserving selectivity. Using implicit toxicity as a stress case, we instantiate this profile on LlavaGuard and Qwen3.5 with sparse feature decompositions. LlavaGuard admits localized handles with a narrow benign-preserving intervention range and modest downstream safety gains, whereas Qwen3.5 supports strong representation-level readout but no comparable selective-control regime under the tested operators. These results show that internal readout and controllability can diverge. Future multimodal safety benchmarks should therefore report not only behavioral safety metrics, but also whether safety-relevant internal signals can be intervention-tested and controlled within a validated operating range.