# Adversarial Confusion Attack: Disrupting Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2511.20494 · HTML: https://mlsanity.com/paper/2511.20494/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 20 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 18 of 18)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [Delving into Transferable Adversarial Examples and Black-box Attacks](https://mlsanity.com/paper/1611.02770/) (2016)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Visual Adversarial Examples Jailbreak Aligned Large Language Models](https://mlsanity.com/paper/2306.13213/) (2023)
- [Image Hijacks: Adversarial Images can Control Generative Models at Runtime](https://mlsanity.com/paper/2309.00236/) (2023)
- [Rethinking Model Ensemble in Transfer-based Adversarial Attacks](https://mlsanity.com/paper/2303.09105/) (2023)
- [WebInject: Prompt Injection Attack to Web Agents](https://mlsanity.com/paper/2505.11717/) (2025)
- [Synthesizing Robust Adversarial Examples](https://mlsanity.com/paper/1707.07397/) (2017)
- [ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents](https://mlsanity.com/paper/2509.16645/) (2025)
- [Transferable Adversarial Attacks on Black-Box Vision-Language Models](https://mlsanity.com/paper/2505.01050/) (2025)
- [Scaling Laws for Black box Adversarial Attacks](https://mlsanity.com/paper/2411.16782/) (2024)
- [Universal Adversarial Attack on Aligned Multimodal LLMs](https://mlsanity.com/paper/2502.07987/) (2025)
- [Environmental Injection Attacks against GUI Agents in Realistic Dynamic Environments](https://mlsanity.com/paper/2509.11250/) (2025)
- [Safety and Security Analysis of Large Language Models: Benchmarking Risk Profile and Harm Potential](https://mlsanity.com/paper/2509.10655/) (2025)
- [Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision-Language Agents](https://mlsanity.com/paper/2510.07809/) (2025)
- [VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models](https://mlsanity.com/paper/2509.25533/) (2025)

## Built on by (most cited first, 3 of 3)

- [Steerability of Instrumental-Convergence Tendencies in LLMs](https://mlsanity.com/paper/2601.01584/) (2026)
- [Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction](https://mlsanity.com/paper/2604.07914/) (2026)
- [Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection](https://mlsanity.com/paper/2609.04533/) (2026)

## Abstract (arXiv)

We introduce the Adversarial Confusion Attack, a new class of threats against multimodal large language models (MLLMs). Unlike jailbreaks or targeted misclassification, the goal is to induce systematic disruption that makes the model generate incoherent or confidently incorrect outputs. Practical applications include embedding such adversarial images into websites to prevent MLLM-powered AI Agents from operating reliably. The proposed attack maximizes next-token entropy using a small ensemble of open-source MLLMs. In the white-box setting, we show that a single adversarial image can disrupt all models in the ensemble, both in the full-image and Adversarial CAPTCHA settings. Despite relying on a basic adversarial technique (PGD), the attack generates perturbations that transfer to both unseen open-source (e.g., Qwen3-VL) and proprietary (e.g., GPT-5.1) models.