# Document Screenshot Retrievers are Vulnerable to Pixel Poisoning Attacks

arXiv: https://arxiv.org/abs/2501.16902 · HTML: https://mlsanity.com/paper/2501.16902/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2025.

## Built on by (most cited first, 2 of 2)

- [One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Image](https://mlsanity.com/paper/2504.02132/) (2025)
- [The Vulnerability of LLM Rankers to Prompt Injection Attacks](https://mlsanity.com/paper/2602.16752/) (2026)

## Abstract (arXiv)

Recent advancements in dense retrieval have introduced vision-language model (VLM)-based retrievers, such as DSE and ColPali, which leverage document screenshots embedded as vectors to enable effective search and offer a simplified pipeline over traditional text-only methods. In this study, we propose three pixel poisoning attack methods designed to compromise VLM-based retrievers and evaluate their effectiveness under various attack settings and parameter configurations. Our empirical results demonstrate that injecting even a single adversarial screenshot into the retrieval corpus can significantly disrupt search results, poisoning the top-10 retrieved documents for 41.9% of queries in the case of DSE and 26.4% for ColPali. These vulnerability rates notably exceed those observed with equivalent attacks on text-only retrievers. Moreover, when targeting a small set of known queries, the attack success rate raises, achieving complete success in certain cases. By exposing the vulnerabilities inherent in vision-language models, this work highlights the potential risks associated with their deployment.