# Natural Color Fool: Towards Boosting Black-box Unrestricted Attacks

arXiv: https://arxiv.org/abs/2210.02041 · HTML: https://mlsanity.com/paper/2210.02041/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 74 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 30 of 52)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser](https://mlsanity.com/paper/1712.02976/) (2017)
- [Feature Denoising for Improving Adversarial Robustness](https://mlsanity.com/paper/1812.03411/) (2018)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)

## Built on by (most cited first, 30 of 38)

- [Diffusion Models for Imperceptible and Transferable Adversarial Attack](https://mlsanity.com/paper/2305.08192/) (2023)
- [Content-based Unrestricted Adversarial Attack](https://mlsanity.com/paper/2305.10665/) (2023)
- [Structure Invariant Transformation for better Adversarial Transferability](https://mlsanity.com/paper/2309.14700/) (2023)
- [CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via Adversarial Latent Search](https://mlsanity.com/paper/2306.10008/) (2023)
- [Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization](https://mlsanity.com/paper/2405.15356/) (2024)
- [Transferable Adversarial Facial Images for Privacy Protection](https://mlsanity.com/paper/2408.01428/) (2024)
- [ADBM: Adversarial diffusion bridge model for reliable adversarial purification](https://mlsanity.com/paper/2408.00315/) (2024)
- [Practical Deep Dispersed Watermarking with Synchronization and Fusion](https://mlsanity.com/paper/2310.14532/) (2023)
- [ASAM: Boosting Segment Anything Model with Adversarial Tuning](https://mlsanity.com/paper/2405.00256/) (2024)
- [Rethinking Impersonation and Dodging Attacks on Face Recognition Systems](https://mlsanity.com/paper/2401.08903/) (2024)
- [Makeup-Guided Facial Privacy Protection via Untrained Neural Network Priors](https://mlsanity.com/paper/2408.12387/) (2024)
- [AdvAD: Exploring Non-Parametric Diffusion for Imperceptible Adversarial Attacks](https://mlsanity.com/paper/2503.09124/) (2025)
- [Boosting the Transferability of Adversarial Examples via Local Mixup and Adaptive Step Size](https://mlsanity.com/paper/2401.13205/) (2024)
- [NatADiff: Adversarial Boundary Guidance for Natural Adversarial Diffusion](https://mlsanity.com/paper/2505.20934/) (2025)
- [Constructing Semantics-Aware Adversarial Examples with a Probabilistic Perspective](https://mlsanity.com/paper/2306.00353/) (2023)
- [Adversarial Attacks on Both Face Recognition and Face Anti-spoofing Models](https://mlsanity.com/paper/2405.16940/) (2024)
- [Improving the Transferability of Adversarial Attacks on Face Recognition with Diverse Parameters Augmentation](https://mlsanity.com/paper/2411.15555/) (2024)
- [Beyond Vulnerabilities: A Survey of Adversarial Attacks as Both Threats and Defenses in Computer Vision Systems](https://mlsanity.com/paper/2508.01845/) (2025)
- [Any Target Can be Offense: Adversarial Example Generation via Generalized Latent Infection](https://mlsanity.com/paper/2407.12292/) (2024)
- [ReToMe-VA: Recursive Token Merging for Video Diffusion-based Unrestricted Adversarial Attack](https://mlsanity.com/paper/2408.05479/) (2024)
- [SCA: Improve Semantic Consistent in Unrestricted Adversarial Attacks via DDPM Inversion](https://mlsanity.com/paper/2410.02240/) (2024)
- [Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment](https://mlsanity.com/paper/2506.01511/) (2025)
- [Defensive Adversarial CAPTCHA: A Semantics-Driven Framework for Natural Adversarial Example Generation](https://mlsanity.com/paper/2506.10685/) (2025)
- [ScoreAdv: Score-based Targeted Generation of Natural Adversarial Examples via Diffusion Models](https://mlsanity.com/paper/2507.06078/) (2025)
- [Adversarial-Guided Diffusion for Multimodal LLM Attacks](https://mlsanity.com/paper/2507.23202/) (2025)
- [Rethinking the Threat and Accessibility of Adversarial Attacks against Face Recognition Systems](https://mlsanity.com/paper/2407.08514/) (2024)
- [Concept-based Adversarial Attack: a Probabilistic Perspective](https://mlsanity.com/paper/2507.02965/) (2025)
- [CertMask: Certifiable Defense Against Adversarial Patches via Theoretically Optimal Mask Coverage](https://mlsanity.com/paper/2511.09834/) (2025)
- [Spatial-Frequency Discriminability for Revealing Adversarial Perturbations](https://mlsanity.com/paper/2305.10856/) (2023)
- [Latent Code Augmentation Based on Stable Diffusion for Data-free Substitute Attacks](https://mlsanity.com/paper/2307.12872/) (2023)

## Abstract (arXiv)

Unrestricted color attacks, which manipulate semantically meaningful color of an image, have shown their stealthiness and success in fooling both human eyes and deep neural networks. However, current works usually sacrifice the flexibility of the uncontrolled setting to ensure the naturalness of adversarial examples. As a result, the black-box attack performance of these methods is limited. To boost transferability of adversarial examples without damaging image quality, we propose a novel Natural Color Fool (NCF) which is guided by realistic color distributions sampled from a publicly available dataset and optimized by our neighborhood search and initialization reset. By conducting extensive experiments and visualizations, we convincingly demonstrate the effectiveness of our proposed method. Notably, on average, results show that our NCF can outperform state-of-the-art approaches by 15.0%~32.9% for fooling normally trained models and 10.0%~25.3% for evading defense methods. Our code is available at https://github.com/ylhz/Natural-Color-Fool.