# Reproducing HotFlip for Corpus Poisoning Attacks in Dense Retrieval

arXiv: https://arxiv.org/abs/2501.04802 · HTML: https://mlsanity.com/paper/2501.04802/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 8 Jan 2025.

## Builds on (most cited first, 27 of 27)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples](https://mlsanity.com/paper/1605.07277/) (2016)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Poisoning Retrieval Corpora by Injecting Adversarial Passages](https://mlsanity.com/paper/2310.19156/) (2023)
- [How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval](https://mlsanity.com/paper/2302.07452/) (2023)
- [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](https://mlsanity.com/paper/1909.09251/) (2019)
- [PRADA: Practical Black-Box Adversarial Attacks against Neural Ranking Models](https://mlsanity.com/paper/2204.01321/) (2022)
- [Adversarial Semantic Collisions](https://mlsanity.com/paper/2011.04743/) (2020)
- [Gradient-based Analysis of NLP Models is Manipulable](https://mlsanity.com/paper/2010.05419/) (2020)
- [Order-Disorder: Imitation Adversarial Attacks for Black-box Neural Ranking Models](https://mlsanity.com/paper/2209.06506/) (2022)
- [Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Method](https://mlsanity.com/paper/2308.09861/) (2023)
- [Adversarial Sampling and Training for Semi-Supervised Information Retrieval](https://mlsanity.com/paper/1811.04155/) (2018)
- [One word at a time: adversarial attacks on retrieval models](https://mlsanity.com/paper/2008.02197/) (2020)
- [Corpus Poisoning via Approximate Greedy Gradient Descent](https://mlsanity.com/paper/2406.05087/) (2024)
- [Topic-oriented Adversarial Attacks against Black-box Neural Ranking Models](https://mlsanity.com/paper/2304.14867/) (2023)
- [Towards Imperceptible Document Manipulations against Neural Ranking Models](https://mlsanity.com/paper/2305.01860/) (2023)
- [Text Counterfactuals via Latent Optimization and Shapley-Guided Search](https://mlsanity.com/paper/2110.11589/) (2021)
- [Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2407.06992/) (2024)
- [Multi-granular Adversarial Attacks against Black-box Neural Ranking Models](https://mlsanity.com/paper/2404.01574/) (2024)
- [Robustness to Programmable String Transformations via Augmented Abstract Training](https://mlsanity.com/paper/2002.09579/) (2020)
- [Robust Information Retrieval](https://mlsanity.com/paper/2406.08891/) (2024)

## Built on by (most cited first, 7 of 7)

- [Unsupervised Corpus Poisoning Attacks in Continuous Space for Dense Retrieval](https://mlsanity.com/paper/2504.17884/) (2025)
- [Rethinking the Privacy of Text Embeddings: A Reproducibility Study of "Text Embeddings Reveal (Almost) As Much As Text"](https://mlsanity.com/paper/2507.07700/) (2025)
- [On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stability](https://mlsanity.com/paper/2604.16576/) (2026)
- [Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrieval](https://mlsanity.com/paper/2604.23396/) (2026)
- [Hypencoder Revisited: Reproducibility and Analysis of Non-Linear Scoring for First-Stage Retrieval](https://mlsanity.com/paper/2604.27037/) (2026)
- [Led to Mislead: Adversarial Content Injection for Attacks on Neural Ranking Models](https://mlsanity.com/paper/2605.01591/) (2026)
- [VerTox: Verifiable Reward-Guided Corpus Poisoning Against Neural Ranking Models](https://mlsanity.com/paper/2609.01325/) (2026)

## Abstract (arXiv)

HotFlip is a topical gradient-based word substitution method for attacking language models. Recently, this method has been further applied to attack retrieval systems by generating malicious passages that are injected into a corpus, i.e., corpus poisoning. However, HotFlip is known to be computationally inefficient, with the majority of time being spent on gradient accumulation for each query-passage pair during the adversarial token generation phase, making it impossible to generate an adequate number of adversarial passages in a reasonable amount of time. Moreover, the attack method itself assumes access to a set of user queries, a strong assumption that does not correspond to how real-world adversarial attacks are usually performed. In this paper, we first significantly boost the efficiency of HotFlip, reducing the adversarial generation process from 4 hours per document to only 15 minutes, using the same hardware. We further contribute experiments and analysis on two additional tasks: (1) transfer-based black-box attacks, and (2) query-agnostic attacks. Whenever possible, we provide comparisons between the original method and our improved version. Our experiments demonstrate that HotFlip can effectively attack a variety of dense retrievers, with an observed trend that its attack performance diminishes against more advanced and recent methods. Interestingly, we observe that while HotFlip performs poorly in a black-box setting, indicating limited capacity for generalization, in query-agnostic scenarios its performance is correlated to the volume of injected adversarial passages.