# PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning

arXiv: https://arxiv.org/abs/2004.05682 · HTML: https://mlsanity.com/paper/2004.05682/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2020.

## Built on by (most cited first, 30 of 67)

- [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://mlsanity.com/paper/2305.12082/) (2023)
- [Are Transformers More Robust Than CNNs?](https://mlsanity.com/paper/2111.05464/) (2021)
- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Improving Transferability of Adversarial Patches on Face Recognition with Generative Models](https://mlsanity.com/paper/2106.15058/) (2021)
- [PatchCleanser: Certifiably Robust Defense against Adversarial Patches for Any Image Classifier](https://mlsanity.com/paper/2108.09135/) (2021)
- [AdvDO: Realistic Adversarial Attacks for Trajectory Prediction](https://mlsanity.com/paper/2209.08744/) (2022)
- [Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving](https://mlsanity.com/paper/2101.06784/) (2021)
- [Simultaneously Optimizing Perturbations and Positions for Black-box Adversarial Patch Attacks](https://mlsanity.com/paper/2212.12995/) (2022)
- [Physically Adversarial Infrared Patches with Learnable Shapes and Locations](https://mlsanity.com/paper/2303.13868/) (2023)
- [PAD: Patch-Agnostic Defense against Adversarial Patch Attacks](https://mlsanity.com/paper/2404.16452/) (2024)
- [Frequency Domain-based Dataset Distillation](https://mlsanity.com/paper/2311.08819/) (2023)
- [ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches](https://mlsanity.com/paper/2203.04412/) (2022)
- [RFLA: A Stealthy Reflected Light Adversarial Attack in the Physical World](https://mlsanity.com/paper/2307.07653/) (2023)
- [ObjectSeeker: Certifiably Robust Object Detection against Patch Hiding Attacks via Patch-agnostic Masking](https://mlsanity.com/paper/2202.01811/) (2022)
- [Efficient Decision-based Black-box Patch Attacks on Video Recognition](https://mlsanity.com/paper/2303.11917/) (2023)
- [Pixle: a fast and effective black-box attack based on rearranging pixels](https://mlsanity.com/paper/2202.02236/) (2022)
- [StyleFool: Fooling Video Classification Systems via Style Transfer](https://mlsanity.com/paper/2203.16000/) (2022)
- [Meta Adversarial Training against Universal Patches](https://mlsanity.com/paper/2101.11453/) (2021)
- [Generative Dynamic Patch Attack](https://mlsanity.com/paper/2111.04266/) (2021)
- [Query-Efficient Decision-based Black-Box Patch Attack](https://mlsanity.com/paper/2307.00477/) (2023)
- [Adversarial Patch Attacks and Defences in Vision-Based Tasks: A Survey](https://mlsanity.com/paper/2206.08304/) (2022)
- [Back in Black: A Comparative Evaluation of Recent State-Of-The-Art Black-Box Attacks](https://mlsanity.com/paper/2109.15031/) (2021)
- [Attacking Video Recognition Models with Bullet-Screen Comments](https://mlsanity.com/paper/2110.15629/) (2021)
- [PointDP: Diffusion-driven Purification against Adversarial Attacks on 3D Point Cloud Recognition](https://mlsanity.com/paper/2208.09801/) (2022)
- [Black-box Targeted Adversarial Attack on Segment Anything (SAM)](https://mlsanity.com/paper/2310.10010/) (2023)
- [Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Feature](https://mlsanity.com/paper/2412.16958/) (2024)
- [Reinforcement Learning Based Sparse Black-box Adversarial Attack on Video Recognition Models](https://mlsanity.com/paper/2108.13872/) (2021)
- [PatchCURE: Improving Certifiable Robustness, Model Utility, and Computation Efficiency of Adversarial Patch Defenses](https://mlsanity.com/paper/2310.13076/) (2023)
- [Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Models](https://mlsanity.com/paper/2406.07145/) (2024)
- [TrajPAC: Towards Robustness Verification of Pedestrian Trajectory Prediction Models](https://mlsanity.com/paper/2308.05985/) (2023)

## Abstract (arXiv)

Patch-based attacks introduce a perceptible but localized change to the input that induces misclassification. A limitation of current patch-based black-box attacks is that they perform poorly for targeted attacks, and even for the less challenging non-targeted scenarios, they require a large number of queries. Our proposed PatchAttack is query efficient and can break models for both targeted and non-targeted attacks. PatchAttack induces misclassifications by superimposing small textured patches on the input image. We parametrize the appearance of these patches by a dictionary of class-specific textures. This texture dictionary is learned by clustering Gram matrices of feature activations from a VGG backbone. PatchAttack optimizes the position and texture parameters of each patch using reinforcement learning. Our experiments show that PatchAttack achieves > 99% success rate on ImageNet for a wide range of architectures, while only manipulating 3% of the image for non-targeted attacks and 10% on average for targeted attacks. Furthermore, we show that PatchAttack circumvents state-of-the-art adversarial defense methods successfully.