# Simultaneously Optimizing Perturbations and Positions for Black-box Adversarial Patch Attacks

arXiv: https://arxiv.org/abs/2212.12995 · HTML: https://mlsanity.com/paper/2212.12995/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 26 Dec 2022.

## Builds on (most cited first, 30 of 34)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [Delving into Transferable Adversarial Examples and Black-box Attacks](https://mlsanity.com/paper/1611.02770/) (2016)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [Black-box Adversarial Attacks with Limited Queries and Information](https://mlsanity.com/paper/1804.08598/) (2018)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [AdvHat: Real-world adversarial attack on ArcFace Face ID system](https://mlsanity.com/paper/1908.08705/) (2019)
- [AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networks](https://mlsanity.com/paper/1805.11770/) (2018)
- [LaVAN: Localized and Visible Adversarial Noise](https://mlsanity.com/paper/1801.02608/) (2018)
- [Adversarial Sticker: A Stealthy Attack Method in the Physical World](https://mlsanity.com/paper/2104.06728/) (2021)
- [Local Gradients Smoothing: Defense against localized adversarial attacks](https://mlsanity.com/paper/1807.01216/) (2018)
- [A General Framework for Adversarial Examples with Objectives](https://mlsanity.com/paper/1801.00349/) (2017)
- [Improving Transferability of Adversarial Patches on Face Recognition with Generative Models](https://mlsanity.com/paper/2106.15058/) (2021)
- [ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization](https://mlsanity.com/paper/1910.06513/) (2019)
- [PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning](https://mlsanity.com/paper/2004.05682/) (2020)
- [Adv-watermark: A Novel Watermark Perturbation for Adversarial Examples](https://mlsanity.com/paper/2008.01919/) (2020)
- [Adversarial Training against Location-Optimized Adversarial Patches](https://mlsanity.com/paper/2005.02313/) (2020)
- [On adversarial patches: real-world attack on ArcFace-100 face recognition system](https://mlsanity.com/paper/1910.07067/) (2019)
- [Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems](https://mlsanity.com/paper/1908.03369/) (2019)
- [Design and Interpretation of Universal Adversarial Patches in Face Detection](https://mlsanity.com/paper/1912.05021/) (2019)

## Built on by (most cited first, 30 of 42)

- [Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption](https://mlsanity.com/paper/2501.10761/) (2025)
- [Physically Adversarial Infrared Patches with Learnable Shapes and Locations](https://mlsanity.com/paper/2303.13868/) (2023)
- [Improving Fast Adversarial Training with Prior-Guided Knowledge](https://mlsanity.com/paper/2304.00202/) (2023)
- [Unified Adversarial Patch for Cross-modal Attacks in the Physical World](https://mlsanity.com/paper/2307.07859/) (2023)
- [Towards Transferable Targeted 3D Adversarial Attack in the Physical World](https://mlsanity.com/paper/2312.09558/) (2023)
- [RFLA: A Stealthy Reflected Light Adversarial Attack in the Physical World](https://mlsanity.com/paper/2307.07653/) (2023)
- [Unified Adversarial Patch for Visible-Infrared Cross-modal Attacks in the Physical World](https://mlsanity.com/paper/2307.14682/) (2023)
- [Physical Adversarial Attacks For Camera-based Smart Systems: Current Trends, Categorization, Applications, Research Challenges, and Future Outlook](https://mlsanity.com/paper/2308.06173/) (2023)
- [Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation](https://mlsanity.com/paper/2306.16170/) (2023)
- [Certified Zeroth-order Black-Box Defense with Robust UNet Denoiser](https://mlsanity.com/paper/2304.06430/) (2023)
- [Adversarial Examples in the Physical World: A Survey](https://mlsanity.com/paper/2311.01473/) (2023)
- [Boosting Adversarial Transferability with Learnable Patch-wise Masks](https://mlsanity.com/paper/2306.15931/) (2023)
- [Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation](https://mlsanity.com/paper/2312.05508/) (2023)
- [CBA: Contextual Background Attack against Optical Aerial Detection in the Physical World](https://mlsanity.com/paper/2302.13519/) (2023)
- [Improving Viewpoint Robustness for Visual Recognition via Adversarial Training](https://mlsanity.com/paper/2307.11528/) (2023)
- [Sequential Manipulation Against Rank Aggregation: Theory and Algorithm](https://mlsanity.com/paper/2407.01916/) (2024)
- [Towards Physically Realizable Adversarial Attacks in Embodied Vision Navigation](https://mlsanity.com/paper/2409.10071/) (2024)
- [A Comprehensive Study on the Robustness of Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarking](https://mlsanity.com/paper/2306.12111/) (2023)
- [Real-world Adversarial Defense against Patch Attacks based on Diffusion Model](https://mlsanity.com/paper/2409.09406/) (2024)
- [Revisiting the Trade-off between Accuracy and Robustness via Weight Distribution of Filters](https://mlsanity.com/paper/2306.03430/) (2023)
- [MVPatch: More Vivid Patch for Adversarial Camouflaged Attacks on Object Detectors in the Physical World](https://mlsanity.com/paper/2312.17431/) (2023)
- [A Survey on Physical Adversarial Attacks against Face Recognition Systems](https://mlsanity.com/paper/2410.16317/) (2024)
- [DynamicPAE: Generating Scene-Aware Physical Adversarial Examples in Real-Time](https://mlsanity.com/paper/2412.08053/) (2024)
- [AdvReal: Physical Adversarial Patch Generation Framework for Security Evaluation of Object Detection Systems](https://mlsanity.com/paper/2505.16402/) (2025)
- [Distributional Modeling for Location-Aware Adversarial Patches](https://mlsanity.com/paper/2306.16131/) (2023)
- [DIP-Watermark: A Double Identity Protection Method Based on Robust Adversarial Watermark](https://mlsanity.com/paper/2404.14693/) (2024)
- [Attack Anything: Blind DNNs via Universal Background Adversarial Attack](https://mlsanity.com/paper/2409.00029/) (2024)
- [Defending Adversarial Patches via Joint Region Localizing and Inpainting](https://mlsanity.com/paper/2307.14242/) (2023)
- [MAGIC: Mastering Physical Adversarial Generation in Context through Collaborative LLM Agents](https://mlsanity.com/paper/2412.08014/) (2024)
- [ErasableMask: A Robust and Erasable Privacy Protection Scheme against Black-box Face Recognition Models](https://mlsanity.com/paper/2412.17038/) (2024)

## Abstract (arXiv)

Adversarial patch is an important form of real-world adversarial attack that brings serious risks to the robustness of deep neural networks. Previous methods generate adversarial patches by either optimizing their perturbation values while fixing the pasting position or manipulating the position while fixing the patch's content. This reveals that the positions and perturbations are both important to the adversarial attack. For that, in this paper, we propose a novel method to simultaneously optimize the position and perturbation for an adversarial patch, and thus obtain a high attack success rate in the black-box setting. Technically, we regard the patch's position, the pre-designed hyper-parameters to determine the patch's perturbations as the variables, and utilize the reinforcement learning framework to simultaneously solve for the optimal solution based on the rewards obtained from the target model with a small number of queries. Extensive experiments are conducted on the Face Recognition (FR) task, and results on four representative FR models show that our method can significantly improve the attack success rate and query efficiency. Besides, experiments on the commercial FR service and physical environments confirm its practical application value. We also extend our method to the traffic sign recognition task to verify its generalization ability.