mlsanityData to 14 Oct 2026

V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs

arXiv 2511.20223 ↗First posted 25 Nov 2025cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Adversarial attacks have evolved from simply disrupting predictions on conventional task-specific models to the more complex goal of manipulating image semantics on Large Vision-Language Models (LVLMs). However, existing methods struggle with controllability and fail to precisely manipulate the semantics of specific concepts in the image. We attribute this limitation to semantic entanglement in the patch-token representations on which adversarial attacks typically operate: global context aggregated by self-attention in the vision encoder dominates individual patch features, making them unreliable handles for precise local semantic manipulation. Our systematic investigation reveals a key insight: value features (V) computed within the transformer attention block serve as much more precise handles for manipulation. We show that V suppresses global-context channels, allowing it to retain high-entropy, disentangled local semantic information. Building on this discovery, we propose V-Attack, a novel method designed for precise local semantic attacks. V-Attack targets the value features and introduces two core components: (1) a Self-Value Enhancement module to refine V's intrinsic semantic richness, and (2) a Text-Guided Value Manipulation module that leverages text prompts to locate source concept and optimize it toward a target concept. By bypassing the entangled patch features, V-Attack achieves highly effective semantic control. Extensive experiments across diverse LVLMs, including LLaVA, InternVL, DeepseekVL and GPT-4o, show that V-Attack improves the attack success rate by an average of 36% over state-of-the-art methods, exposing critical vulnerabilities in modern visual-language understanding. Our code and data are available https://github.com/Summu77/V-Attack.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 60 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

49 of 60 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuning22 May
  2. 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o113 Mar
  3. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations14 Feb
  4. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  5. 2024Adversarial Robustness for Visual Grounding of Multimodal Large Language Models16 May
  6. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models16 Apr
  7. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  8. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  9. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  10. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks6 Dec
  11. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  12. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
2 more
  1. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  2. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2023Vision Transformers Need Registerscited by 871
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
23 more
  1. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  2. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  3. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
  4. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  5. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  6. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  7. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  8. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  9. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  10. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  11. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  12. 2022Frequency Domain Model Augmentation for Adversarial Attackcited by 130
  13. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 113
  14. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  15. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  16. 2023Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllabilitycited by 66
  17. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 43
  18. 2020Transferable, Controllable, and Inconspicuous Adversarial Attacks on Person Re-identification With Deep Mis-Rankingcited by 42
  19. 2022Query-Efficient Black-box Adversarial Attacks Guided by a Transfer-based Priorcited by 38
  20. 2024Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Imagecited by 26
  21. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25
  22. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Promptscited by 18
  23. 2022Bootstrap Generalization Ability from Loss Landscape Perspectivecited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026On the Adversarial Robustness of Multimodal LLM Judgescited by 3
  2. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  3. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  4. 2026Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijackingcited by 0
  5. 2026REALM: A Unified Red-Teaming Benchmark for Physical-World VLMscited by 0
  6. 20263D FaceShell: Attribute Transfer in 3D Face Avatars as a VLM Defense Mechanismcited by 0
  7. 2026QR-Structured Thermal Triggers for Targeted Semantic Attacks on Infrared Vision-Language Modelscited by 0
  8. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0
  9. 2026It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at ε≤ 4/255cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models
  2. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  3. 2026PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attention
  4. 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models
  5. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
  6. 2026Breaking the weakest link to evade vision language models
  7. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
  8. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models