mlsanityData to 13 Oct 2026

Adversarial Confusion Attack: Disrupting Multimodal Large Language Models

arXiv 2511.20494 ↗First posted 25 Nov 2025cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

We introduce the Adversarial Confusion Attack, a new class of threats against multimodal large language models (MLLMs). Unlike jailbreaks or targeted misclassification, the goal is to induce systematic disruption that makes the model generate incoherent or confidently incorrect outputs. Practical applications include embedding such adversarial images into websites to prevent MLLM-powered AI Agents from operating reliably. The proposed attack maximizes next-token entropy using a small ensemble of open-source MLLMs. In the white-box setting, we show that a single adversarial image can disrupt all models in the ensemble, both in the full-image and Adversarial CAPTCHA settings. Despite relying on a basic adversarial technique (PGD), the attack generates perturbations that transfer to both unseen open-source (e.g., Qwen3-VL) and proprietary (e.g., GPT-5.1) models.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 20 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

18 of 20 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents20 Sep
  2. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models2 May
  3. 2025Universal Adversarial Attack on Aligned Multimodal LLMs11 Feb
  4. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 14

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,379
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,953
  3. 2016Universal adversarial perturbationscited by 782
  4. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  5. 2024The Platonic Representation Hypothesiscited by 369
  6. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 170
  7. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 89
  8. 2025WebInject: Prompt Injection Attack to Web Agentscited by 50
  9. 2017Synthesizing Robust Adversarial Examplescited by 39
  10. 2024Scaling Laws for Black box Adversarial Attackscited by 8
  11. 2025Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environmentscited by 6
  12. 2025Safety and Security Analysis of Large Language Models: Benchmarking Risk Profile and Harm Potentialcited by 3
2 more
  1. 2025Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision-Language Agentscited by 3
  2. 2025VISOR++: Universal Visual Inputs based Steering for Large Vision Language Modelscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Steerability of Instrumental-Convergence Tendencies in LLMscited by 1
  2. 2026Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reductioncited by 0
  3. 2026Repeat-After-Me: Black-Box Adaptive Visual Prompt Injectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey
  2. 2025Survey of Adversarial Robustness in Multimodal Large Language Models
  3. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs
  4. 2025Towards Robust Multimodal Large Language Models Against Jailbreak Attacks
  5. 2025Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities
  6. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  7. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Security
  8. 2026Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domain