# An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferability

arXiv: https://arxiv.org/abs/2308.02897 · HTML: https://mlsanity.com/paper/2308.02897/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 38 references that are in the map.
- First posted on arXiv 5 Aug 2023.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [Delving into Transferable Adversarial Examples and Black-box Attacks](https://mlsanity.com/paper/1611.02770/) (2016)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks](https://mlsanity.com/paper/1704.01155/) (2017)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser](https://mlsanity.com/paper/1712.02976/) (2017)
- [Attacks Which Do Not Kill Training Make Adversarial Learning Stronger](https://mlsanity.com/paper/2002.11242/) (2020)
- [Improving Adversarial Robustness via Promoting Ensemble Diversity](https://mlsanity.com/paper/1901.08846/) (2019)
- [A Self-supervised Approach for Adversarial Robustness](https://mlsanity.com/paper/2006.04924/) (2020)
- [Feature Importance-aware Transferable Adversarial Attacks](https://mlsanity.com/paper/2107.14185/) (2021)
- [ComDefend: An Efficient Image Compression Model to Defend Adversarial Examples](https://mlsanity.com/paper/1811.12673/) (2018)
- [Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examples](https://mlsanity.com/paper/1803.05787/) (2018)
- [Improving Adversarial Transferability via Neuron Attribution-Based Attacks](https://mlsanity.com/paper/2204.00008/) (2022)
- [Curriculum Adversarial Training](https://mlsanity.com/paper/1805.04807/) (2018)
- [Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferability](https://mlsanity.com/paper/2111.10752/) (2021)
- [Patch-wise Attack for Fooling Deep Neural Network](https://mlsanity.com/paper/2007.06765/) (2020)
- [DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles](https://mlsanity.com/paper/2009.14720/) (2020)

## Built on by (most cited first, 30 of 42)

- [A Survey on Transferability of Adversarial Examples across Deep Neural Networks](https://mlsanity.com/paper/2310.17626/) (2023)
- [AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2511.12149/) (2025)
- [As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?](https://mlsanity.com/paper/2403.12693/) (2024)
- [Rethinking Impersonation and Dodging Attacks on Face Recognition Systems](https://mlsanity.com/paper/2401.08903/) (2024)
- [Scaling Laws for Black box Adversarial Attacks](https://mlsanity.com/paper/2411.16782/) (2024)
- [PubDef: Defending Against Transfer Attacks From Public Models](https://mlsanity.com/paper/2310.17645/) (2023)
- [Boosting the Transferability of Adversarial Examples via Local Mixup and Adaptive Step Size](https://mlsanity.com/paper/2401.13205/) (2024)
- [Understanding Model Ensemble in Transferable Adversarial Attack](https://mlsanity.com/paper/2410.06851/) (2024)
- [Adversarial Example Soups: Improving Transferability and Stealthiness for Free](https://mlsanity.com/paper/2402.18370/) (2024)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)
- [Exploring Adversarial Attacks against Latent Diffusion Model from the Perspective of Adversarial Transferability](https://mlsanity.com/paper/2401.07087/) (2024)
- [Adversarial Attacks on Both Face Recognition and Face Anti-spoofing Models](https://mlsanity.com/paper/2405.16940/) (2024)
- [Improving the Transferability of Adversarial Attacks on Face Recognition with Diverse Parameters Augmentation](https://mlsanity.com/paper/2411.15555/) (2024)
- [SlowPerception: Physical-World Latency Attack against Visual Perception in Autonomous Driving](https://mlsanity.com/paper/2406.05800/) (2024)
- [Effective Black-Box Multi-Faceted Attacks Breach Vision Large Language Model Guardrails](https://mlsanity.com/paper/2502.05772/) (2025)
- [TF-Attack: Transferable and Fast Adversarial Attacks on Large Language Models](https://mlsanity.com/paper/2408.13985/) (2024)
- [Improving Fast Adversarial Training via Self-Knowledge Guidance](https://mlsanity.com/paper/2409.17589/) (2024)
- [Use as Many Surrogates as You Want: Selective Ensemble Attack to Unleash Transferability without Sacrificing Resource Efficiency](https://mlsanity.com/paper/2505.12644/) (2025)
- [SoK: Analyzing Adversarial Examples: A Framework to Study Adversary Knowledge](https://mlsanity.com/paper/2402.14937/) (2024)
- [Learning to Learn Transferable Generative Attack for Person Re-Identification](https://mlsanity.com/paper/2409.04208/) (2024)
- [Enhancing Adversarial Transferability via Component-Wise Transformation](https://mlsanity.com/paper/2501.11901/) (2025)
- [Understanding Cross-Model Perceptual Invariances Through Ensemble Metamers](https://mlsanity.com/paper/2504.01739/) (2025)
- [Boosting Adversarial Transferability via Residual Perturbation Attack](https://mlsanity.com/paper/2508.05689/) (2025)
- [ViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformers](https://mlsanity.com/paper/2508.12384/) (2025)
- [OTI: A Model-free and Visually Interpretable Measure of Image Attackability](https://mlsanity.com/paper/2601.17536/) (2026)
- [The Power of Decaying Steps: Enhancing Attack Stability and Transferability for Sign-based Optimizers](https://mlsanity.com/paper/2602.19096/) (2026)
- [ADAPT to Robustify Prompt Tuning Vision Transformers](https://mlsanity.com/paper/2403.13196/) (2024)
- [Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective](https://mlsanity.com/paper/2408.03944/) (2024)
- [Sustainable Self-evolution Adversarial Training](https://mlsanity.com/paper/2412.02270/) (2024)
- [FSPGD: Rethinking Black-box Attacks on Semantic Segmentation](https://mlsanity.com/paper/2502.01262/) (2025)

## Abstract (arXiv)

While the transferability property of adversarial examples allows the adversary to perform black-box attacks (i.e., the attacker has no knowledge about the target model), the transfer-based adversarial attacks have gained great attention. Previous works mostly study gradient variation or image transformations to amplify the distortion on critical parts of inputs. These methods can work on transferring across models with limited differences, i.e., from CNNs to CNNs, but always fail in transferring across models with wide differences, such as from CNNs to ViTs. Alternatively, model ensemble adversarial attacks are proposed to fuse outputs from surrogate models with diverse architectures to get an ensemble loss, making the generated adversarial example more likely to transfer to other models as it can fool multiple models concurrently. However, existing ensemble attacks simply fuse the outputs of the surrogate models evenly, thus are not efficacious to capture and amplify the intrinsic transfer information of adversarial examples. In this paper, we propose an adaptive ensemble attack, dubbed AdaEA, to adaptively control the fusion of the outputs from each model, via monitoring the discrepancy ratio of their contributions towards the adversarial objective. Furthermore, an extra disparity-reduced filter is introduced to further synchronize the update direction. As a result, we achieve considerable improvement over the existing ensemble attacks on various datasets, and the proposed AdaEA can also boost existing transfer-based attacks, which further demonstrates its efficacy and versatility.