mlsanityData to 14 Oct 2026

A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1

arXiv 2503.10635 ↗First posted 13 Mar 2025cs.CV cs.AI cs.LGcited by 37

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Despite promising performance on open-source large vision-language models (LVLMs), transfer-based targeted attacks often fail against closed-source commercial LVLMs. Analyzing failed adversarial perturbations reveals that the learned perturbations typically originate from a uniform distribution and lack clear semantic details, resulting in unintended responses. This critical absence of semantic information leads commercial black-box LVLMs to either ignore the perturbation entirely or misinterpret its embedded semantics, thereby causing the attack to fail. To overcome these issues, we propose to refine semantic clarity by encoding explicit semantic details within local regions, thus ensuring the capture of finer-grained features and inter-model transferability, and by concentrating modifications on semantically rich areas rather than applying them uniformly. To achieve this, we propose a simple yet highly effective baseline: at each optimization step, the adversarial image is cropped randomly by a controlled aspect ratio and scale, resized, and then aligned with the target image in the embedding space. While the naive source-target matching method has been utilized before in the literature, we are the first to provide a tight analysis, which establishes a close connection between perturbation optimization and semantics. Experimental results confirm our hypothesis. Our adversarial examples crafted with local-aggregated perturbations focused on crucial regions exhibit surprisingly good transferability to commercial LVLMs, including GPT-4.5, GPT-4o, Gemini-2.0-flash, Claude-3.5/3.7-sonnet, and even reasoning models like o1, Claude-3.7-thinking and Gemini-2.0-flash-thinking. Our approach achieves success rates exceeding 90% on GPT-4.5, 4o, and o1, significantly outperforming all prior state-of-the-art attack methods with lower ℓ1/ℓ2 perturbations.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 48 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

35 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models16 Apr
  2. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  3. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  11. 2016Adversarial examples in the physical worldcited by 1,769
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
20 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  2. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  3. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
  4. 2023GPTScore: Evaluate as You Desirecited by 361
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  6. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  7. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  8. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  9. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  10. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  11. 2022Frequency Domain Model Augmentation for Adversarial Attackcited by 130
  12. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 89
  13. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  14. 2023Cross-Modal Retrieval: A Systematic Review of Methods and Future Directionscited by 40
  15. 2022Query-Efficient Black-box Adversarial Attacks Guided by a Transfer-based Priorcited by 38
  16. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25
  17. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Promptscited by 18
  18. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Modelscited by 18
  19. 2022Bootstrap Generalization Ability from Loss Landscape Perspectivecited by 16
  20. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?cited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 28.

37 papers.

Sort:

By area:

  1. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 43
  2. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  3. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMscited by 9
  4. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 8
  5. 2025Black-box Optimization of LLM Outputs by Asking for Directionscited by 7
  6. 2026Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targetingcited by 6
  7. 2025GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbationscited by 5
  8. 2025PhysPatch: A Physically Realizable and Transferable Adversarial Patch Attack for Multimodal Large Language Models-based Autonomous Driving Systemscited by 4
  9. 2026Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Modelscited by 4
  10. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Modelscited by 4
  11. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 3
  12. 2025Disrupting Hierarchical Reasoning: Adversarial Protection for Geographic Privacy in Multimodal Reasoning Modelscited by 3
25 more
  1. 2026PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attentioncited by 3
  2. 2026On the Adversarial Robustness of Multimodal LLM Judgescited by 3
  3. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  4. 2026Frequency-Domain Regularized Adversarial Alignment for Transferable Attacks against Closed-Source MLLMscited by 2
  5. 2026Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agentscited by 2
  6. 2026Universal Adversarial Attacks against Closed-Source MLLMs via Target-View Routed Meta Optimizationcited by 1
  7. 2026VALD: Multi-Stage Vision Attack Detection for Efficient LVLM Defensecited by 1
  8. 2026When Background Matters: Breaking Medical Vision Language Models by Transferable Attackcited by 1
  9. 2026Adversarial Attacks Against MLLMs via Progressive Resolution Processing and Adaptive Feature Alignmentcited by 1
  10. 2026Feature-Space Smoothing: Certified Robustness of Deep Representationscited by 0
  11. 2026SGHA-Attack: Semantic-Guided Hierarchical Alignment for Transferable Targeted Attacks on Vision-Language Modelscited by 0
  12. 2026Stage-wise Attention-Guided Region Sequencing for Adversarial Attacks on Large Vision-Language Modelscited by 0
  13. 2026Covert Visual Prompt Injection against Commercial Multimodal Large Language Modelscited by 0
  14. 2026Mosaic: Multimodal Jailbreak against Closed-Source VLMs via Multi-View Ensemble Optimizationcited by 0
  15. 2026Laundering AI Authority with Adversarial Examplescited by 0
  16. 2026DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Modelscited by 0
  17. 2026REALM: A Unified Red-Teaming Benchmark for Physical-World VLMscited by 0
  18. 20263D FaceShell: Attribute Transfer in 3D Face Avatars as a VLM Defense Mechanismcited by 0
  19. 2026The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrailscited by 0
  20. 2026Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domaincited by 0
  21. 2026Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Modelscited by 0
  22. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0
  23. 2026It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at ε≤ 4/255cited by 0
  24. 2026Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMscited by 0
  25. 2026GeoCloak: An Agentic Framework for Semantics-Preserving Geolocation Privacy via Peripheral Clue Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  2. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
  3. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
  4. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
  5. 2025Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector
  6. 2026Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization
  7. 2026Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction
  8. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models