# T-SEA: Transfer-based Self-Ensemble Attack on Object Detection

arXiv: https://arxiv.org/abs/2211.09773 · HTML: https://mlsanity.com/paper/2211.09773/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks](https://mlsanity.com/paper/1908.06281/) (2019)
- [Adversarial Examples for Semantic Segmentation and Object Detection](https://mlsanity.com/paper/1703.08603/) (2017)
- [Fooling automated surveillance cameras: adversarial patches to attack person detection](https://mlsanity.com/paper/1904.08653/) (2019)
- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detector](https://mlsanity.com/paper/1804.05810/) (2018)
- [Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectors](https://mlsanity.com/paper/1910.14667/) (2019)
- [Transferable Adversarial Attacks for Image and Video Object Detection](https://mlsanity.com/paper/1811.12641/) (2018)
- [Towards Practical Certifiable Patch Defense with Vision Transformer](https://mlsanity.com/paper/2203.08519/) (2022)
- [Robust Adversarial Perturbation on Deep Proposal-based Models](https://mlsanity.com/paper/1809.05962/) (2018)
- [Object Hider: Adversarial Patch Attack Against Object Detectors](https://mlsanity.com/paper/2010.14974/) (2020)
- [DPAttack: Diffused Patch Attacks against Universal Object Detection](https://mlsanity.com/paper/2010.11679/) (2020)
- [RPATTACK: Refined Patch Attack on General Object Detectors](https://mlsanity.com/paper/2103.12469/) (2021)
- [Efficient universal shuffle attack for visual object tracking](https://mlsanity.com/paper/2203.06898/) (2022)

## Built on by (most cited first, 30 of 59)

- [How Robust is Google's Bard to Adversarial Image Attacks?](https://mlsanity.com/paper/2309.11751/) (2023)
- [Rethinking Model Ensemble in Transfer-based Adversarial Attacks](https://mlsanity.com/paper/2303.09105/) (2023)
- [A Survey on Transferability of Adversarial Examples across Deep Neural Networks](https://mlsanity.com/paper/2310.17626/) (2023)
- [SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation](https://mlsanity.com/paper/2312.04913/) (2023)
- [PAD: Patch-Agnostic Defense against Adversarial Patch Attacks](https://mlsanity.com/paper/2404.16452/) (2024)
- [Boosting the Transferability of Adversarial Attacks with Global Momentum Initialization](https://mlsanity.com/paper/2211.11236/) (2022)
- [Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey](https://mlsanity.com/paper/2410.23687/) (2024)
- [Efficient Decision-based Black-box Patch Attacks on Video Recognition](https://mlsanity.com/paper/2303.11917/) (2023)
- [DUAW: Data-free Universal Adversarial Watermark against Stable Diffusion Customization](https://mlsanity.com/paper/2308.09889/) (2023)
- [NumbOD: A Spatial-Frequency Fusion Attack Against Object Detectors](https://mlsanity.com/paper/2412.16955/) (2024)
- [Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Feature](https://mlsanity.com/paper/2412.16958/) (2024)
- [When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2511.21192/) (2025)
- [As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?](https://mlsanity.com/paper/2403.12693/) (2024)
- [Diffusion to Confusion: Naturalistic Adversarial Patch Generation Based on Diffusion Model for Object Detector](https://mlsanity.com/paper/2307.08076/) (2023)
- [Adversarial Examples in the Physical World: A Survey](https://mlsanity.com/paper/2311.01473/) (2023)
- [UV-Attack: Physical-World Adversarial Attacks for Person Detection via Dynamic-NeRF-based UV Mapping](https://mlsanity.com/paper/2501.05783/) (2025)
- [Scaling Laws for Black box Adversarial Attacks](https://mlsanity.com/paper/2411.16782/) (2024)
- [TASAR: Transfer-based Attack on Skeletal Action Recognition](https://mlsanity.com/paper/2409.02483/) (2024)
- [CDUPatch: Color-Driven Universal Adversarial Patch Attack for Dual-Modal Visible-Infrared Detectors](https://mlsanity.com/paper/2504.10888/) (2025)
- [A Comprehensive Study on the Robustness of Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarking](https://mlsanity.com/paper/2306.12111/) (2023)
- [AdvLogo: Adversarial Patch Attack against Object Detectors based on Diffusion Models](https://mlsanity.com/paper/2409.07002/) (2024)
- [MVPatch: More Vivid Patch for Adversarial Camouflaged Attacks on Object Detectors in the Physical World](https://mlsanity.com/paper/2312.17431/) (2023)
- [BadPatch: Diffusion-Based Generation of Physical Adversarial Patches](https://mlsanity.com/paper/2412.01440/) (2024)
- [DynamicPAE: Generating Scene-Aware Physical Adversarial Examples in Real-Time](https://mlsanity.com/paper/2412.08053/) (2024)
- [AdvReal: Physical Adversarial Patch Generation Framework for Security Evaluation of Object Detection Systems](https://mlsanity.com/paper/2505.16402/) (2025)
- [SlowPerception: Physical-World Latency Attack against Visual Perception in Autonomous Driving](https://mlsanity.com/paper/2406.05800/) (2024)
- [PBCAT: Patch-based composite adversarial training against physically realizable attacks on object detection](https://mlsanity.com/paper/2506.23581/) (2025)
- [AFN: Adaptive Fusion Normalization via an Encoder-Decoder Framework](https://mlsanity.com/paper/2308.03321/) (2023)
- [Use as Many Surrogates as You Want: Selective Ensemble Attack to Unleash Transferability without Sacrificing Resource Efficiency](https://mlsanity.com/paper/2505.12644/) (2025)
- [Revisiting Adversarial Patch Defenses on Object Detectors: Unified Evaluation, Large-Scale Dataset, and New Insights](https://mlsanity.com/paper/2508.00649/) (2025)

## Abstract (arXiv)

Compared to query-based black-box attacks, transfer-based black-box attacks do not require any information of the attacked models, which ensures their secrecy. However, most existing transfer-based approaches rely on ensembling multiple models to boost the attack transferability, which is time- and resource-intensive, not to mention the difficulty of obtaining diverse models on the same task. To address this limitation, in this work, we focus on the single-model transfer-based black-box attack on object detection, utilizing only one model to achieve a high-transferability adversarial attack on multiple black-box detectors. Specifically, we first make observations on the patch optimization process of the existing method and propose an enhanced attack framework by slightly adjusting its training strategies. Then, we analogize patch optimization with regular model optimization, proposing a series of self-ensemble approaches on the input data, the attacked model, and the adversarial patch to efficiently make use of the limited information and prevent the patch from overfitting. The experimental results show that the proposed framework can be applied with multiple classical base attack methods (e.g., PGD and MIM) to greatly improve the black-box transferability of the well-optimized patch on multiple mainstream detectors, meanwhile boosting white-box performance. Our code is available at https://github.com/VDIGPKU/T-SEA.