# The Uncontrolled Variable: Vision-Language Model Refusal Responds to Image Presence in Ways Risk Cannot Explain

arXiv: https://arxiv.org/abs/2609.26174 · HTML: https://mlsanity.com/paper/2609.26174/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 16 of 16)

- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion](https://mlsanity.com/paper/2403.07865/) (2024)
- [Understanding and Rectifying Safety Perception Distortion in VLMs](https://mlsanity.com/paper/2502.13095/) (2025)
- [Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training](https://mlsanity.com/paper/2502.11455/) (2025)
- [Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study](https://mlsanity.com/paper/2505.15389/) (2025)
- [DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Prompt](https://mlsanity.com/paper/2506.09353/) (2025)
- [Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents](https://mlsanity.com/paper/2602.16931/) (2026)
- [Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack](https://mlsanity.com/paper/2505.21967/) (2025)
- [Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift](https://mlsanity.com/paper/2603.17372/) (2026)
- [SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues](https://mlsanity.com/paper/2603.19092/) (2026)
- [Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images](https://mlsanity.com/paper/2603.08486/) (2026)
- [Exposing LLM Safety Gaps Through Mathematical Encoding:New Attacks and Systematic Analysis](https://mlsanity.com/paper/2605.03441/) (2026)
- [Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility](https://mlsanity.com/paper/2602.03402/) (2026)
- [Benign-to-Toxic Jailbreaking: Inducing Harmful Responses from Harmless Prompts](https://mlsanity.com/paper/2505.21556/) (2025)

## Abstract (arXiv)

We show that aligned vision-language models also condition refusal on a property of a request's form: whether an image is attached, holding everything the request asks fixed. Attaching a blank canvas, an image that cannot be read, cannot relate to the request, and is byte-identical across every prompt in its condition, shifts benign refusal by tens of points. The shift is not blanket caution but a threshold shift: genuinely neutral instructions are almost unaffected (<=2 percentage points on three of four hosted models) while borderline-benign prompts move +23 to +51 points, so the cost falls on sensitivity-adjacent traffic, meaning benign questions about privacy, self-harm, violence and illegal activity. There is a benign reading of such a threshold, namely that attachment correlates with risk in real traffic, and we take it seriously; a black-box study cannot measure that correlation and we do not claim to. What it can test is whether the response to attachment is robust to variation carrying no information about the request, and on four independent measurements it is not. It varies with canvas colour and pixel count. Its sign inverts across checkpoints. It survives an explicit instruction to disregard the image. And on one model it fires on a bare assertion that an attachment exists, with nothing attached and the modality word contributing none of it. Finally we price it. On a matched harmful set the same canvas does lower attack success, so the cue buys something. But the charge is decoupled from the purchase: the checkpoint with the least harmful headroom we measure, completing only 2% of plain harmful requests, still pays the benign cost in full, and across our models the harmful-side denominator falls as alignment improves while the benign cost does not track it down. Image presence is not a conservative default that a deployer chose and priced. It is an uncontrolled variable.