mlsanityData to 14 Oct 2026

Decoy Images Amplify Caption-Mediated Defenses Against Encoded Jailbreaks

arXiv 2608.01043 ↗First posted 2 Aug 2026cs.CR cs.AIcited by 0

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

We report a counter-intuitive interaction between image inputs and existing black-box defenses on Vision--Language Models (VLMs): pairing an encoded jailbreak prompt with an unrelated decoy image can sharply lower attack success rate (ASR). The operative change is in the defense pipeline, not in the image. Across five frontier VLMs, two encoded-attack families, and three black-box defenses, a caption-mediated defense (ECSO) that leaves ASR essentially unchanged on text-only encoded input drops it by up to 73pp once a content-free decoy is attached; every non-saturated contrast is significant under exact McNemar tests. We advance two hypotheses for this pattern, supported by indirect evidence rather than pipeline introspection, since a black-box threat model precludes inspecting vendor internals: caption-mediated defenses branch on image presence, and intrinsic image-side safety engages on image-resident content. Three controls constrain the explanation. Blank-canvas and natural-photograph decoys reproduce the effect on every model, implicating image presence rather than content; the effect replicates on three open-weight VLMs served with no moderation layer, so it is not a vendor-filtering artifact; and a non-symbolic, meaning-based encoder reproduces it, so it is not specific to symbolic obfuscation. Attaching a decoy unconditionally is not deployable --- it raises benign refusal to 20--79%, an inflation of +10 to +67pp --- but gating attachment on a lightweight encoded-input detector returns benign refusal to the text baseline while preserving the safety gain wherever the detector fires, making detector recall the binding constraint. Under adaptive attacks that target the caption-mediated re-check, the effect degrades but holds. We frame this as an observation about pipeline interaction, not as a robust defense.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 12 of its 20 references that are in the map. First posted on arXiv 2 Aug 2026.

Builds on

12 of 20 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowcharts28 Feb
  2. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks23 Nov
  3. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 9

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
  3. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 76
  4. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 44
  5. 2024Jailbreaking Large Language Models with Symbolic Mathematicscited by 16
  6. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 13
  7. 2025DETAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention Modificationcited by 10
  8. 2025Why Not Act on What You Know? Unleashing Safety Potential of LLMs via Self-Aware Guard Enhancementcited by 8
  9. 2026Exposing LLM Safety Gaps Through Mathematical Encoding:New Attacks and Systematic Analysiscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Which Image Property Carries the Jailbreak? A Controlled Dissection of Image-to-Text Jailbreaks
  2. 2026When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness?
  3. 2026Reading Between the Pixels: An Inscriptive Jailbreak Attack on Text-to-Image Models
  4. 2025TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models
  5. 2026Fully Unleashing the Multimodal Attacker: Meta-Adaptive Jailbreaking of Vision-Language Models
  6. 2026Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filters
  7. 2025Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense
  8. 2026Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models