It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at ε≤ 4/255
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
Abstract (arXiv)
Vision Language Models (VLMs) are widely deployed in safety-critical scenarios, and understanding to which extent they can be controlled by adversarial perturbation is a prerequisite for evaluating their trustworthiness. Existing representation-alignment attacks, which make a VLM perceive a target image, achieve limited success at varepsilon ≤ 4/255. Therefore, VLMs seems robust to perturbations in this range. We show that this robustness does not hold, as targeted semantic substitution succeeds within the same range. Specifically, we align each stream of the source image with its counterpart in the target image in the victim VLM's post-merger token space, operating under a white-box threat model. We evaluate under a strict success criterion, requiring the model to simultaneously name the target, confirm its presence, and deny the source. In images, target semantics appear at varepsilon = 2/255 and complete replacement reaches 38% at varepsilon = 4/255. On video, complete replacement reaches 35.9% at varepsilon = 1/255. We also observe a phenomenon of semantic fusion, where Large Language Model (LLM) rationalizes contradictory visual signals into a coherent narrative.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 36 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
35 of 36 references.
Closest ancestors · 18
Same direction, newest first
- 2026Breaking the weakest link to evade vision language models19 Aug
- 2026PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention23 Feb
- 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs25 Nov
- 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents20 Sep
- 2025Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector19 Aug
- 2025CAVALRY-V: A Large-Scale Generator Framework for Adversarial Attacks on Video MLLMs1 Jul
- 2025Attention! Your Vision Language Model Could Be Maliciously Manipulated26 May
- 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models23 May
- 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o113 Mar
- 2025Transferability of Adversarial Attacks in Video-based MLLMs: A Cross-modal Image-to-Video Approach2 Jan
- 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models11 Dec
- 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks24 Nov
6 more
- 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models9 Oct
- 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models7 Oct
- 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models4 Dec
- 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
- 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
- 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
Other references · 17
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
- 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
- 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
- 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
- 2023Are aligned neural networks adversarially aligned?cited by 305
- 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
- 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
- 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
5 more
- 2015Adversarial Manipulation of Deep Representationscited by 56
- 2024LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Modelscited by 49
- 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 42
- 2024FMM-Attack: A Flow-based Multi-modal Adversarial Attack on Video-based LLMscited by 14
- 2024Video Watermarking: Safeguarding Your Video from (Unauthorized) Annotations by Video-based LLMscited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SGHA-Attack: Semantic-Guided Hierarchical Alignment for Transferable Targeted Attacks on Vision-Language Models
- 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
- 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
- 2026Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
- 2026Single-Sample Black-Box Membership Inference Attack against Vision-Language Models via Cross-modal Semantic Alignment
- 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
- 2025Blind Spot Navigation: Evolutionary Discovery of Sensitive Semantic Concepts for LVLMs
- 2024Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models