mlsanityData to 14 Oct 2026

Multimodal Resource-Exhaustion Attacks on Vision-Language Models via Joint Pixel-Prompt Optimization

arXiv 2609.05889 ↗First posted 5 Sep 2026cs.AI cs.CR cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Resource-exhaustion attacks against autoregressive vision-language models (VLMs) typically assume unimodal threat models, treating the image branch as the primary optimization surface while holding user-visible prompts fixed. Even recent loop-centric variants remain confined to this single-channel paradigm, leaving the exploitation of availability unexplored as a cross-modal optimization problem over jointly controllable input surfaces. We introduce Joint Pixel-Prompt Optimization (JPPO), the first compound adversarial framework elevating the visible prompt to a first-class adversarial variable alongside image perturbations. Under a restricted joint-input threat model, JPPO performs coupled, stagewise optimization over both the pixel and prompt surfaces. This produces synergistic cost amplification, mechanistically distinct from loop-dependent failures, exhibiting negligible loop incidence in our experiments. Evaluating five open-source VLM families on MS COCO and ImageNet under an 8/255 infinity-norm budget, JPPO achieves over 4.6x latency and 5.3x energy amplification on Qwen2.5-VL-7B, and over 36.6x latency with 32.7x energy amplification on BLIP-2. This represents the strongest cost amplification among directly compared baselines while requiring substantially fewer optimization iterations. Ablations confirm this amplification arises from multimodal coordination rather than prompt length or isolated modalities. These findings reveal structural blind spots in current VLM serving defenses, motivating cost-aware robustness evaluation as a first-class security requirement for multimodal deployments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 42 references that are in the map. First posted on arXiv 5 Sep 2026.

Builds on

17 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves29 Oct
  2. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images20 Jan
  3. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  4. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  5. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 12

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2018Sanity Checks for Saliency Mapscited by 1,156
  7. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  8. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  9. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 172
  10. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  11. 2020Sponge Examples: Energy-Latency Attacks on Neural Networkscited by 111
  12. 2022Energy-Latency Attacks via Sponge Poisoningcited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization
  2. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  3. 2026Breaking the weakest link to evade vision language models
  4. 2026GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models
  5. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
  6. 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models
  7. 2025An Image Is Worth Ten Thousand Words: Verbose-Text Induction Attacks on VLMs
  8. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Models