# Defense of Adversarial Ranking Attack in Text Retrieval: Benchmark and Baseline via Detection

arXiv: https://arxiv.org/abs/2307.16816 · HTML: https://mlsanity.com/paper/2307.16816/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [PRADA: Practical Black-Box Adversarial Attacks against Neural Ranking Models](https://mlsanity.com/paper/2204.01321/) (2022)
- [Adversarial Semantic Collisions](https://mlsanity.com/paper/2011.04743/) (2020)
- [Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models](https://mlsanity.com/paper/2209.06506/) (2022)
- [Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators](https://mlsanity.com/paper/2111.13057/) (2021)
- [One word at a time: adversarial attacks on retrieval models](https://mlsanity.com/paper/2008.02197/) (2020)
- [Dealing with Typos for BERT-based Passage Retrieval and Ranking](https://mlsanity.com/paper/2108.12139/) (2021)
- [Analysing the Robustness of Dual Encoders for Dense Retrieval Against Misspellings](https://mlsanity.com/paper/2205.02303/) (2022)
- [CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typos](https://mlsanity.com/paper/2204.00716/) (2022)
- [Topic-oriented Adversarial Attacks against Black-box Neural Ranking Models](https://mlsanity.com/paper/2304.14867/) (2023)
- [Towards Imperceptible Document Manipulations against Neural Ranking Models](https://mlsanity.com/paper/2305.01860/) (2023)
- [Are Neural Ranking Models Robust?](https://mlsanity.com/paper/2108.05018/) (2021)
- [BERT Rankers are Brittle: a Study using Adversarial Document Perturbations](https://mlsanity.com/paper/2206.11724/) (2022)
- [Contrastive Fine-tuning Improves Robustness for Neural Rankers](https://mlsanity.com/paper/2105.12932/) (2021)
- [Certified Robustness to Word Substitution Ranking Attack for Neural Ranking Models](https://mlsanity.com/paper/2209.06691/) (2022)
- [A Study on FGSM Adversarial Training for Neural Retrieval](https://mlsanity.com/paper/2301.10576/) (2023)
- [TextDefense: Adversarial Text Detection based on Word Importance Entropy](https://mlsanity.com/paper/2302.05892/) (2023)
- [Towards Proactive Information Retrieval in Noisy Text with Wikipedia Concepts](https://mlsanity.com/paper/2210.09877/) (2022)

## Built on by (most cited first, 3 of 3)

- [Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2407.06992/) (2024)
- [Quantitative Certification of Agentic Tool Selection](https://mlsanity.com/paper/2510.03992/) (2025)
- [RobustMask: Certified Robustness against Adversarial Neural Ranking Attack via Randomized Masking](https://mlsanity.com/paper/2512.23307/) (2025)

## Abstract (arXiv)

Neural ranking models (NRMs) have undergone significant development and have become integral components of information retrieval (IR) systems. Unfortunately, recent research has unveiled the vulnerability of NRMs to adversarial document manipulations, potentially exploited by malicious search engine optimization practitioners. While progress in adversarial attack strategies aids in identifying the potential weaknesses of NRMs before their deployment, the defensive measures against such attacks, like the detection of adversarial documents, remain inadequately explored. To mitigate this gap, this paper establishes a benchmark dataset to facilitate the investigation of adversarial ranking defense and introduces two types of detection tasks for adversarial documents. A comprehensive investigation of the performance of several detection baselines is conducted, which involve examining the spamicity, perplexity, and linguistic acceptability, and utilizing supervised classifiers. Experimental results demonstrate that a supervised classifier can effectively mitigate known attacks, but it performs poorly against unseen attacks. Furthermore, such classifier should avoid using query text to prevent learning the classification on relevance, as it might lead to the inadvertent discarding of relevant documents.