# QAIR: Practical Query-efficient Black-Box Attacks for Image Retrieval

arXiv: https://arxiv.org/abs/2103.02927 · HTML: https://mlsanity.com/paper/2103.02927/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 4 Mar 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detector](https://mlsanity.com/paper/1804.05810/) (2018)
- [Query-Efficient Hard-label Black-box Attack:An Optimization-based Approach](https://mlsanity.com/paper/1807.04457/) (2018)
- [Improving Black-box Adversarial Attacks with a Transfer-based Prior](https://mlsanity.com/paper/1906.06919/) (2019)
- [Sign-OPT: A Query-Efficient Hard-label Adversarial Attack](https://mlsanity.com/paper/1909.10773/) (2019)
- [MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient Estimation](https://mlsanity.com/paper/2005.03161/) (2020)
- [DaST: Data-free Substitute Training for Adversarial Attacks](https://mlsanity.com/paper/2003.12703/) (2020)
- [Guessing Smart: Biased Sampling for Efficient Black-Box Adversarial Attacks](https://mlsanity.com/paper/1812.09803/) (2018)
- [Transferable, Controllable, and Inconspicuous Adversarial Attacks on Person Re-identification With Deep Mis-Ranking](https://mlsanity.com/paper/2004.04199/) (2020)
- [Curls & Whey: Boosting Black-Box Adversarial Attacks](https://mlsanity.com/paper/1904.01160/) (2019)
- [Subspace Attack: Exploiting Promising Subspaces for Query-Efficient Black-box Attacks](https://mlsanity.com/paper/1906.04392/) (2019)
- [Universal Perturbation Attack Against Image Retrieval](https://mlsanity.com/paper/1812.00552/) (2018)
- [Learning Black-Box Attackers with Transferable Priors and Query Feedback](https://mlsanity.com/paper/2010.11742/) (2020)
- [Adversarial Ranking Attack and Defense](https://mlsanity.com/paper/2002.11293/) (2020)
- [Web-Scale Responsive Visual Search at Bing](https://mlsanity.com/paper/1802.04914/) (2018)

## Built on by (most cited first, 22 of 22)

- [Frequency Domain Model Augmentation for Adversarial Attack](https://mlsanity.com/paper/2207.05382/) (2022)
- [Advances in adversarial attacks and defenses in computer vision: A survey](https://mlsanity.com/paper/2108.00401/) (2021)
- [Adversarial Sticker: A Stealthy Attack Method in the Physical World](https://mlsanity.com/paper/2104.06728/) (2021)
- [Person Re-identification Method Based on Color Attack and Joint Defence](https://mlsanity.com/paper/2111.09571/) (2021)
- [A Large-scale Multiple-objective Method for Black-box Attack against Object Detection](https://mlsanity.com/paper/2209.07790/) (2022)
- [Beyond ImageNet Attack: Towards Crafting Adversarial Examples for Black-box Domains](https://mlsanity.com/paper/2201.11528/) (2022)
- [Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models](https://mlsanity.com/paper/2209.06506/) (2022)
- [DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization](https://mlsanity.com/paper/2408.11071/) (2024)
- [A Survey of Robustness and Safety of 2D and 3D Deep Learning Models Against Adversarial Attacks](https://mlsanity.com/paper/2310.00633/) (2023)
- [Enhancing Adversarial Robustness for Deep Metric Learning](https://mlsanity.com/paper/2203.01439/) (2022)
- [Practical Relative Order Attack in Deep Ranking](https://mlsanity.com/paper/2103.05248/) (2021)
- [Adversarial Attacks on Video Object Segmentation with Hard Region Discovery](https://mlsanity.com/paper/2309.13857/) (2023)
- [Adversarial Attack and Defense in Deep Ranking](https://mlsanity.com/paper/2106.03614/) (2021)
- [Meta Generative Attack on Person Reidentification](https://mlsanity.com/paper/2301.06286/) (2023)
- [Poisoning Prompt-Guided Sampling in Video Large Language Models](https://mlsanity.com/paper/2509.20851/) (2025)
- [Collapse-Aware Triplet Decoupling for Adversarially Robust Image Retrieval](https://mlsanity.com/paper/2312.07364/) (2023)
- [Provable Robust Saliency-based Explanations](https://mlsanity.com/paper/2212.14106/) (2022)
- [ScAR: Scaling Adversarial Robustness for LiDAR Object Detection](https://mlsanity.com/paper/2312.03085/) (2023)
- [A Survey of Social Cybersecurity: Techniques for Attack Detection, Evaluations, Challenges, and Future Prospects](https://mlsanity.com/paper/2504.04311/) (2025)
- [Adversarial Video Promotion Against Text-to-Video Retrieval](https://mlsanity.com/paper/2508.06964/) (2025)
- [Unsupervised Multi-Criteria Adversarial Detection in Deep Image Retrieval](https://mlsanity.com/paper/2304.04228/) (2023)
- [Robust Ranking Explanations](https://mlsanity.com/paper/2307.04024/) (2023)

## Abstract (arXiv)

We study the query-based attack against image retrieval to evaluate its robustness against adversarial examples under the black-box setting, where the adversary only has query access to the top-k ranked unlabeled images from the database. Compared with query attacks in image classification, which produce adversaries according to the returned labels or confidence score, the challenge becomes even more prominent due to the difficulty in quantifying the attack effectiveness on the partial retrieved list. In this paper, we make the first attempt in Query-based Attack against Image Retrieval (QAIR), to completely subvert the top-k retrieval results. Specifically, a new relevance-based loss is designed to quantify the attack effects by measuring the set similarity on the top-k retrieval results before and after attacks and guide the gradient optimization. To further boost the attack efficiency, a recursive model stealing method is proposed to acquire transferable priors on the target model and generate the prior-guided gradients. Comprehensive experiments show that the proposed attack achieves a high attack success rate with few queries against the image retrieval systems under the black-box setting. The attack evaluations on the real-world visual search engine show that it successfully deceives a commercial system such as Bing Visual Search with 98% attack success rate by only 33 queries on average.