# Practical Relative Order Attack in Deep Ranking

arXiv: https://arxiv.org/abs/2103.05248 · HTML: https://mlsanity.com/paper/2103.05248/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 60 references that are in the map.
- First posted on arXiv 9 Mar 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Black-box Adversarial Attacks with Limited Queries and Information](https://mlsanity.com/paper/1804.08598/) (2018)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks](https://mlsanity.com/paper/1904.02884/) (2019)
- [Adversarial Risk and the Dangers of Evaluating Against Weak Attacks](https://mlsanity.com/paper/1802.05666/) (2018)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [Efficient Decision-based Black-box Adversarial Attacks on Face Recognition](https://mlsanity.com/paper/1904.04433/) (2019)
- [Query-Efficient Hard-label Black-box Attack:An Optimization-based Approach](https://mlsanity.com/paper/1807.04457/) (2018)
- [QEBA: Query-Efficient Boundary-Based Blackbox Attack](https://mlsanity.com/paper/2005.14137/) (2020)
- [Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://mlsanity.com/paper/2002.08473/) (2020)
- [NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks](https://mlsanity.com/paper/1905.00441/) (2019)
- [Transferable, Controllable, and Inconspicuous Adversarial Attacks on Person Re-identification With Deep Mis-Ranking](https://mlsanity.com/paper/2004.04199/) (2020)
- [Deep Metric Learning Beyond Binary Supervision](https://mlsanity.com/paper/1904.09626/) (2019)
- [Universal Perturbation Attack Against Image Retrieval](https://mlsanity.com/paper/1812.00552/) (2018)
- [Targeted Mismatch Adversarial Attack: Query with a Flower to Retrieve the Tower](https://mlsanity.com/paper/1908.09163/) (2019)

## Built on by (most cited first, 9 of 9)

- [Person Re-identification Method Based on Color Attack and Joint Defence](https://mlsanity.com/paper/2111.09571/) (2021)
- [Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models](https://mlsanity.com/paper/2209.06506/) (2022)
- [Enhancing Adversarial Robustness for Deep Metric Learning](https://mlsanity.com/paper/2203.01439/) (2022)
- [Adversarial Attack and Defense in Deep Ranking](https://mlsanity.com/paper/2106.03614/) (2021)
- [Collapse-Aware Triplet Decoupling for Adversarially Robust Image Retrieval](https://mlsanity.com/paper/2312.07364/) (2023)
- [Provable Robust Saliency-based Explanations](https://mlsanity.com/paper/2212.14106/) (2022)
- [Adversarial Video Promotion Against Text-to-Video Retrieval](https://mlsanity.com/paper/2508.06964/) (2025)
- [Advancing Deep Metric Learning Through Multiple Batch Norms And Multi-Targeted Adversarial Examples](https://mlsanity.com/paper/2211.16253/) (2022)
- [Robust Ranking Explanations](https://mlsanity.com/paper/2307.04024/) (2023)

## Abstract (arXiv)

Recent studies unveil the vulnerabilities of deep ranking models, where an imperceptible perturbation can trigger dramatic changes in the ranking result. While previous attempts focus on manipulating absolute ranks of certain candidates, the possibility of adjusting their relative order remains under-explored. In this paper, we formulate a new adversarial attack against deep ranking systems, i.e., the Order Attack, which covertly alters the relative order among a selected set of candidates according to an attacker-specified permutation, with limited interference to other unrelated candidates. Specifically, it is formulated as a triplet-style loss imposing an inequality chain reflecting the specified permutation. However, direct optimization of such white-box objective is infeasible in a real-world attack scenario due to various black-box limitations. To cope with them, we propose a Short-range Ranking Correlation metric as a surrogate objective for black-box Order Attack to approximate the white-box method. The Order Attack is evaluated on the Fashion-MNIST and Stanford-Online-Products datasets under both white-box and black-box threat models. The black-box attack is also successfully implemented on a major e-commerce platform. Comprehensive experimental evaluations demonstrate the effectiveness of the proposed methods, revealing a new type of ranking model vulnerability.