# Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization

arXiv: https://arxiv.org/abs/2510.06732 · HTML: https://mlsanity.com/paper/2510.06732/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Optimization-based Prompt Injection Attack to LLM-as-a-Judge](https://mlsanity.com/paper/2403.17710/) (2024)
- [Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels](https://mlsanity.com/paper/2310.14122/) (2023)
- [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://mlsanity.com/paper/2101.05667/) (2021)
- [A Survey on LLM-powered Agents for Recommender Systems](https://mlsanity.com/paper/2502.10050/) (2025)
- [Ranking Manipulation for Conversational Search Engines](https://mlsanity.com/paper/2406.03589/) (2024)
- [CheatAgent: Attacking LLM-Empowered Recommender Systems via LLM Agent](https://mlsanity.com/paper/2504.13192/) (2025)
- [Stealthy Attack on Large Language Model based Recommendation](https://mlsanity.com/paper/2402.14836/) (2024)
- [LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses](https://mlsanity.com/paper/2406.04755/) (2024)
- [StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization](https://mlsanity.com/paper/2504.05804/) (2025)
- [Large Language Model Enhanced Recommender Systems: A Survey](https://mlsanity.com/paper/2412.13432/) (2024)
- [Dynamics of Adversarial Attacks on Large Language Model-Based Search Engines](https://mlsanity.com/paper/2501.00745/) (2025)

## Built on by (most cited first, 9 of 9)

- [FinCARDS: Card-Based Analyst Reranking for Financial Document Question Answering](https://mlsanity.com/paper/2601.06992/) (2026)
- [Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers](https://mlsanity.com/paper/2601.12263/) (2026)
- [Controlling Output Rankings in Generative Engines for LLM-based Search](https://mlsanity.com/paper/2602.03608/) (2026)
- [GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization](https://mlsanity.com/paper/2605.29107/) (2026)
- [Do Vision Language Models Understand Human Engagement in Games?](https://mlsanity.com/paper/2603.18480/) (2026)
- [Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders](https://mlsanity.com/paper/2607.24869/) (2026)
- [Position: Recommender Systems Should Move Beyond Platform-Centric Ranking toward Personal Agent-Mediated Recommendation](https://mlsanity.com/paper/2609.11942/) (2026)
- [The Vulnerability of LLM Rankers to Prompt Injection Attacks](https://mlsanity.com/paper/2602.16752/) (2026)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used as rerankers in information retrieval, yet their ranking behavior can be steered by small, natural-sounding prompts. To expose this vulnerability, we present Rank Anything First (RAF), a two-stage token optimization method that crafts concise textual perturbations to consistently promote a target item in LLM-generated rankings while remaining hard to detect. Stage 1 uses Greedy Coordinate Gradient to shortlist candidate tokens at the current position by combining the gradient of the rank-target with a readability score; Stage 2 evaluates those candidates under exact ranking and readability losses using an entropy-based dynamic weighting scheme, and selects a token via temperature-controlled sampling. RAF generates ranking-promoting prompts token-by-token, guided by dual objectives: maximizing ranking effectiveness and preserving linguistic naturalness. Experiments across multiple LLMs show that RAF significantly boosts the rank of target items using naturalistic language, with greater robustness than existing methods in both promoting target items and maintaining naturalness. These findings underscore a critical security implication: LLM-based reranking is inherently susceptible to adversarial manipulation, raising new challenges for the trustworthiness and robustness of modern retrieval systems. Our code is available at: https://github.com/glad-lab/RAF.