mlsanityData to 7 Oct 2026

Rethinking Model Ensemble in Transfer-based Adversarial Attacks

arXiv 2303.09105 ↗First posted 16 Mar 2023cs.CVcited by 88

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It is widely recognized that deep learning models lack robustness to adversarial examples. An intriguing property of adversarial examples is that they can transfer across different models, which enables black-box attacks without any knowledge of the victim model. An effective strategy to improve the transferability is attacking an ensemble of models. However, previous works simply average the outputs of different models, lacking an in-depth analysis on how and why model ensemble methods can strongly improve the transferability. In this paper, we rethink the ensemble in adversarial attacks and define the common weakness of model ensemble with two properties: 1) the flatness of loss landscape; and 2) the closeness to the local optimum of each model. We empirically and theoretically show that both properties are strongly correlated with the transferability and propose a Common Weakness Attack (CWA) to generate more transferable adversarial examples by promoting these two properties. Experimental results on both image classification and object detection tasks validate the effectiveness of our approach to improving the adversarial transferability, especially when attacking adversarially trained models. We also successfully apply our method to attack a black-box large vision-language model -- Google's Bard, showing the practical effectiveness. Code is available at urlhttps://github.com/huanranchen/AdversarialAttacks.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 72 of its 90 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

72 of 90 references.

References · 72

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  10. 2016Densely Connected Convolutional Networkscited by 6,691
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2022A ConvNet for the 2020scited by 3,545
  6. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2018YOLOv3: An Incremental Improvementcited by 2,652
  9. 2019Searching for MobileNetV3cited by 1,984
  10. 2016Adversarial examples in the physical worldcited by 1,762
  11. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  12. 2016YOLO9000: Better, Faster, Strongercited by 1,429
  13. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  14. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  15. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  16. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  17. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  18. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  19. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  20. 2020Designing Network Design Spacescited by 954
  21. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  22. 2016Adversarial Machine Learning at Scalecited by 934
  23. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  24. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  25. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  26. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  27. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  28. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  29. 2017Countering Adversarial Images using Input Transformationscited by 517
  30. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  31. 2022Diffusion Models for Adversarial Purificationcited by 469
  32. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 440
  33. 2017Mitigating Adversarial Effects Through Randomizationcited by 382
  34. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 352
  35. 2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
  36. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  37. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  38. 2019Fooling automated surveillance cameras: adversarial patches to attack person detectioncited by 268

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 32; 2025: 32; 2026: 15.

88 papers.

Sort:

By area:

  1. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  2. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  3. 2023Robust Classification via a Single Diffusion Modelcited by 79
  4. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  5. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  6. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 42
  7. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 42
  8. 2024Elucidating the Design Space of Dataset Condensationcited by 40
  9. 2024Boosting Jailbreak Attack with Momentumcited by 38
  10. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attackscited by 37
  11. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 36
  12. 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1cited by 34
38 more
  1. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  2. 2024A Transfer Attack to Image Watermarkscited by 26
  3. 2024Your Diffusion Model is Secretly a Certifiably Robust Classifiercited by 24
  4. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  5. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 22
  6. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  7. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
  8. 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agentscited by 19
  9. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Modelscited by 15
  10. 2023Sharpness-Aware Minimization Alone can Improve Adversarial Robustnesscited by 14
  11. 2024Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Modelscited by 14
  12. 2024Exploring the Robustness of In-Context Learning with Noisy Labelscited by 13
  13. 2024Bag of Tricks to Boost Adversarial Transferabilitycited by 12
  14. 2024Adversarial Representation Engineering: A General Model Editing Framework for Large Language Modelscited by 12
  15. 2024DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domaincited by 12
  16. 2025FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEctioncited by 11
  17. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Modelscited by 10
  18. 2024Scaling Laws for Black box Adversarial Attackscited by 8
  19. 2024FaceShield: Defending Facial Image against Deepfake Threatscited by 8
  20. 2025JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Modelscited by 8
  21. 2023PubDef: Defending Against Transfer Attacks From Public Modelscited by 7
  22. 2025Towards the Worst-case Robustness of Large Language Modelscited by 7
  23. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  24. 2024Adversarial Example Soups: Improving Transferability and Stealthiness for Freecited by 6
  25. 2024Boosting Adversarial Transferability with Low-Cost Optimization via Maximin Expected Flatnesscited by 6
  26. 2024Understanding Model Ensemble in Transferable Adversarial Attackcited by 6
  27. 2023A Comprehensive Study on the Robustness of Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarkingcited by 5
  28. 2024Improving the Transferability of Adversarial Examples by Feature Augmentationcited by 5
  29. 2026Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targetingcited by 5
  30. 2024Adversarial Attacks on Both Face Recognition and Face Anti-spoofing Modelscited by 4
  31. 2025Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferabilitycited by 4
  32. 2025TRAP: Targeted Redirecting of Agentic Preferencescited by 4
  33. 2025Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigationcited by 4
  34. 2026Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minimacited by 4
  35. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 3
  36. 2025CAVALRY-V: A Large-Scale Generator Framework for Adversarial Attacks on Video MLLMscited by 3
  37. 2025GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbationscited by 3
  38. 2025HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generationcited by 3

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed
  2. 2024Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness
  3. 2023Improving Adversarial Transferability via Model Alignment
  4. 2023Improving Adversarial Transferability by Stable Diffusion
  5. 2023Adversarial defense based on distribution transfer
  6. 2025Enhancing Adversarial Transferability by Balancing Exploration and Exploitation with Gradient-Guided Sampling
  7. 2024Exploring Adversarial Attacks against Latent Diffusion Model from the Perspective of Adversarial Transferability
  8. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation