mlsanityData to 7 Oct 2026

Semantic Router: On the Feasibility of Hijacking MLLMs via a Single Adversarial Perturbation

arXiv 2511.20002 ↗First posted 25 Nov 2025cs.CV cs.AI cs.CRcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) are increasingly deployed in stateless systems, such as autonomous driving and robotics. This paper investigates a novel threat: Semantic-Aware Hijacking. We explore the feasibility of hijacking multiple stateless decisions simultaneously using a single universal perturbation. We introduce the Semantic-Aware Universal Perturbation (SAUP), which acts as a semantic router, "actively" perceiving input semantics and routing them to distinct, attacker-defined targets. To achieve this, we conduct theoretical and empirical analysis on the geometric properties in the latent space. Guided by these insights, we propose the Semantic-Oriented (SORT) optimization strategy and annotate a new dataset with fine-grained semantics to evaluate performance. Extensive experiments on three representative MLLMs demonstrate the fundamental feasibility of this attack, achieving a 66% attack success rate over five targets using a single frame against Qwen.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 40 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

30 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models7 Oct
  2. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  3. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  4. 2024Test-Time Backdoor Attacks on Multimodal Large Language Models13 Feb
  5. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  6. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  7. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  9. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2016Universal adversarial perturbationscited by 774
11 more
  1. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  2. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 352
  3. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 242
  4. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  5. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  6. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  7. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  8. 2021On Generating Transferable Targeted Perturbationscited by 65
  9. 2025One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIPcited by 15
  10. 2021RamBoAttack: A Robust Query Efficient Deep Neural Network Decision Exploitcited by 11
  11. 2023Multi-attacks: Many images + the same adversarial attack → many target labelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs
  2. 2025Semantic Representation Attack against Aligned Large Language Models
  3. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Security
  4. 2026Text-Anchored Semantic Perturbations for Transferable Jailbreak Attacks on Multimodal Large Language Models
  5. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  6. 2025Phi: Preference Hijacking in Multi-modal Large Language Models at Inference Time
  7. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Driving
  8. 2026Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades