mlsanityData to 14 Oct 2026

A Cross-Architecture Audit of Direction-Based Inference-Time Defences in Vision-Language Models

arXiv 2607.27910 ↗First posted 30 Jul 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Inference time defences against vision language model jailbreaks often subtract a calibrated direction from the residual stream at a chosen decoder layer. We compare five defence candidates across 15 model and layer cells from four architectural families under a magnitude controlled protocol that matches the intervention size for each prompt and pairs every direction with a random control of the same norm. The candidates are the mean image conditioning shift, a CMRM style refusal direction, a ShiftDC style attack specific residual, a prompt instruction to ignore the image, and a random control. No single candidate dominates on both refusal recovery and utility preservation. The image conditioning shift leads on LLaVA 1.5 and Pixtral 12B and is the only candidate whose utility loss remains at the measurement noise floor in every family. The prompt instruction leads on Qwen2.5 VL, while the attack specific residual leads on Qwen2 VL 2B. The image conditioning direction is direction specific in 13 of 15 cells, but strongly architecture specific and nontransferable across the only dimension compatible pair, LLaVA 1.5 13B and Pixtral 12B. We also connect text only and multimodal refusal geometry. The CMRM direction has positive cosine alignment with the image conditioning shift in all 15 cells, with mean 0.35, range 0.17 to 0.65, 15 to 25 times the random vector null, and a sign test p value of about 3e-5. These results show that the two recipes recover partially overlapping geometry and that direction based defences should be calibrated separately for each language decoder family.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 12 of its 15 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

12 of 15 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift18 Mar
  2. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring12 Dec
  3. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States20 Feb
  4. 2025Understanding and Rectifying Safety Perception Distortion in VLMs18 Feb
  5. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap14 Feb
  6. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  7. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 5

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  2. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  3. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  4. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 41
  5. 2024Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teamingcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
  2. 2026FLIP: Final Layer Inference-Time Probing for Vision-Language Models
  3. 2026Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
  4. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  5. 2026Closing the Activation-Cone Blind Spot: Response-Time Probing and Unified Defense
  6. 2026Architectural Sampling: Test-Time Scaling via Computational Diversity in Frozen Vision-Language Models
  7. 2026Selective Channel Restoration for Backdoored Vision-Language Models
  8. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models