# Universal Perturbation Attack Against Image Retrieval

arXiv: https://arxiv.org/abs/1812.00552 · HTML: https://mlsanity.com/paper/1812.00552/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2018.

## Built on by (most cited first, 30 of 32)

- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Frequency-driven Imperceptible Adversarial Attack on Semantic Similarity](https://mlsanity.com/paper/2203.05151/) (2022)
- [Deep Learning for Instance Retrieval: A Survey](https://mlsanity.com/paper/2101.11282/) (2021)
- [A Survey On Universal Adversarial Attack](https://mlsanity.com/paper/2103.01498/) (2021)
- [CMUA-Watermark: A Cross-Model Universal Adversarial Watermark for Combating Deepfakes](https://mlsanity.com/paper/2105.10872/) (2021)
- [Person Re-identification Method Based on Color Attack and Joint Defence](https://mlsanity.com/paper/2111.09571/) (2021)
- [Cross-Modality Perturbation Synergy Attack for Person Re-identification](https://mlsanity.com/paper/2401.10090/) (2024)
- [Test-Time Backdoor Attacks on Multimodal Large Language Models](https://mlsanity.com/paper/2402.08577/) (2024)
- [QAIR: Practical Query-efficient Black-Box Attacks for Image Retrieval](https://mlsanity.com/paper/2103.02927/) (2021)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [Attack to Fool and Explain Deep Networks](https://mlsanity.com/paper/2106.10606/) (2021)
- [RetrievalGuard: Provably Robust 1-Nearest Neighbor Image Retrieval](https://mlsanity.com/paper/2206.11225/) (2022)
- [Cross-Modality Attack Boosted by Gradient-Evolutionary Multiform Optimization](https://mlsanity.com/paper/2409.17977/) (2024)
- [Enhancing Adversarial Robustness for Deep Metric Learning](https://mlsanity.com/paper/2203.01439/) (2022)
- [Practical Relative Order Attack in Deep Ranking](https://mlsanity.com/paper/2103.05248/) (2021)
- [Prototype-supervised Adversarial Network for Targeted Attack of Deep Hashing](https://mlsanity.com/paper/2105.07553/) (2021)
- [Adversarial Attack and Defense in Deep Ranking](https://mlsanity.com/paper/2106.03614/) (2021)
- [Discriminator-Free Generative Adversarial Attack](https://mlsanity.com/paper/2107.09225/) (2021)
- [RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models](https://mlsanity.com/paper/2304.10727/) (2023)
- [Subverting Fair Image Search with Generative Adversarial Perturbations](https://mlsanity.com/paper/2205.02414/) (2022)
- [Meta Generative Attack on Person Reidentification](https://mlsanity.com/paper/2301.06286/) (2023)
- [MINIMAL: Mining Models for Data Free Universal Adversarial Triggers](https://mlsanity.com/paper/2109.12406/) (2021)
- [Feature Extraction Matters More: Universal Deepfake Disruption through Attacking Ensemble Feature Extractors](https://mlsanity.com/paper/2303.00200/) (2023)
- [Collapse-Aware Triplet Decoupling for Adversarially Robust Image Retrieval](https://mlsanity.com/paper/2312.07364/) (2023)
- [Learning to Learn Transferable Generative Attack for Person Re-Identification](https://mlsanity.com/paper/2409.04208/) (2024)
- [A Survey of Social Cybersecurity: Techniques for Attack Detection, Evaluations, Challenges, and Future Prospects](https://mlsanity.com/paper/2504.04311/) (2025)
- [Towards Effective Multi-Label Recognition Attacks via Knowledge Graph Consistency](https://mlsanity.com/paper/2207.05137/) (2022)
- [Look Closer to Your Enemy: Learning to Attack via Teacher-Student Mimicking](https://mlsanity.com/paper/2207.13381/) (2022)
- [Towards Robust Universal Perturbation Attacks: A Float-Coded, Penalty-Driven Evolutionary Approach](https://mlsanity.com/paper/2601.12624/) (2026)
- [Adversarial Attacks on Robot Localization Systems via Deep Feature Perturbation](https://mlsanity.com/paper/2606.01892/) (2026)

## Abstract (arXiv)

Universal adversarial perturbations (UAPs), a.k.a. input-agnostic perturbations, has been proved to exist and be able to fool cutting-edge deep learning models on most of the data samples. Existing UAP methods mainly focus on attacking image classification models. Nevertheless, little attention has been paid to attacking image retrieval systems. In this paper, we make the first attempt in attacking image retrieval systems. Concretely, image retrieval attack is to make the retrieval system return irrelevant images to the query at the top ranking list. It plays an important role to corrupt the neighbourhood relationships among features in image retrieval attack. To this end, we propose a novel method to generate retrieval-against UAP to break the neighbourhood relationships of image features via degrading the corresponding ranking metric. To expand the attack method to scenarios with varying input sizes or untouchable network parameters, a multi-scale random resizing scheme and a ranking distillation strategy are proposed. We evaluate the proposed method on four widely-used image retrieval datasets, and report a significant performance drop in terms of different metrics, such as mAP and mP@10. Finally, we test our attack methods on the real-world visual search engine, i.e., Google Images, which demonstrates the practical potentials of our methods.