mlsanityData to 7 Oct 2026

Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting

arXiv 2602.17645 ↗First posted 19 Feb 2026cs.LG cs.AI cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Black-box adversarial attacks on Large Vision-Language Models (LVLMs) are challenging due to missing gradients and complex multimodal boundaries. While prior state-of-the-art transfer-based approaches like M-Attack perform well using local crop-level matching between source and target images, we find this induces high-variance, nearly orthogonal gradients across iterations, violating coherent local alignment and destabilizing optimization. We attribute this to (i) ViT translation sensitivity that yields spike-like gradients and (ii) structural asymmetry between source and target crops. We reformulate local matching as an asymmetric expectation over source transformations and target semantics, and build a gradient-denoising upgrade to M-Attack. On the source side, Multi-Crop Alignment (MCA) averages gradients from multiple independently sampled local views per iteration to reduce variance. On the target side, Auxiliary Target Alignment (ATA) replaces aggressive target augmentation with a small auxiliary set from a semantically correlated distribution, producing a smoother, lower-variance target manifold. We further reinterpret momentum as Patch Momentum, replaying historical crop gradients; combined with a refined patch-size ensemble (PE+), this strengthens transferable directions. Together these modules form M-Attack-V2, a simple, modular enhancement over M-Attack that substantially improves transfer-based black-box attacks on frontier LVLMs: boosting success rates on Claude-4.0 from 8% to 30%, Gemini-2.5-Pro from 83% to 97%, and GPT-5 from 98% to 100%, outperforming prior black-box LVLM attacks. Code and data are publicly available at: https://github.com/vila-lab/M-Attack-V2.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 41 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

32 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o113 Mar
  2. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models16 Apr
  3. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2016Adversarial examples in the physical worldcited by 1,762
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  12. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
16 more
  1. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  2. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  3. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  5. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  6. 2022Frequency Domain Model Augmentation for Adversarial Attackcited by 128
  7. 2018Adversarial Attacks and Defences Competitioncited by 96
  8. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  9. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 88
  10. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 42
  11. 2023Cross-Modal Retrieval: A Systematic Review of Methods and Future Directionscited by 40
  12. 2022Query-Efficient Black-box Adversarial Attacks Guided by a Transfer-based Priorcited by 38
  13. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25
  14. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Promptscited by 18
  15. 2022Bootstrap Generalization Ability from Loss Landscape Perspectivecited by 16
  16. 2017PatternNet: Visual Pattern Mining with Deep Neural Networkcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMscited by 1
  2. 2026Frequency-Domain Regularized Adversarial Alignment for Transferable Attacks against Closed-Source MLLMscited by 1
  3. 2026Adversarial Attacks Against MLLMs via Progressive Resolution Processing and Adaptive Feature Alignmentcited by 0
  4. 2026Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domaincited by 0
  5. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SGHA-Attack: Semantic-Guided Hierarchical Alignment for Transferable Targeted Attacks on Vision-Language Models
  2. 2025Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector
  3. 2026Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs
  4. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  5. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
  6. 2026Universal Adversarial Attacks against Closed-Source MLLMs via Target-View Routed Meta Optimization
  7. 2026Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization
  8. 2026Structure-Guided Visual Perturbation Neutralization for LVLMs