# Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders

arXiv: https://arxiv.org/abs/2607.24869 · HTML: https://mlsanity.com/paper/2607.24869/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Builds on (most cited first, 20 of 20)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [Unbiased Learning-to-Rank with Biased Feedback](https://mlsanity.com/paper/1608.04468/) (2016)
- [RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!](https://mlsanity.com/paper/2312.02724/) (2023)
- [Split and Merge: Aligning Position Biases in LLM-based Evaluators](https://mlsanity.com/paper/2310.01432/) (2023)
- [Unbiased Learning to Rank with Unbiased Propensity Estimation](https://mlsanity.com/paper/1804.05938/) (2018)
- [ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrieval](https://mlsanity.com/paper/2402.15838/) (2024)
- [LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking](https://mlsanity.com/paper/2406.00231/) (2024)
- [Evaluating Position Bias in Large Language Model Recommendations](https://mlsanity.com/paper/2508.02020/) (2025)
- [StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization](https://mlsanity.com/paper/2504.05804/) (2025)
- [Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset](https://mlsanity.com/paper/2404.02543/) (2024)
- [Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization](https://mlsanity.com/paper/2510.06732/) (2025)
- [Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Models](https://mlsanity.com/paper/2412.18770/) (2024)
- [One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation](https://mlsanity.com/paper/2604.27599/) (2026)
- [LLM-based Listwise Reranking under the Effect of Positional Bias](https://mlsanity.com/paper/2604.03642/) (2026)
- [SimAug: Enhancing Recommendation with Pretrained Language Models for Dense and Balanced Data Augmentation](https://mlsanity.com/paper/2505.01695/) (2025)

## Built on by (most cited first, 1 of 1)

- [SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agents](https://mlsanity.com/paper/2609.35889/) (2026)

## Abstract (arXiv)

Large language models (LLMs) used as listwise rerankers in recommendation systems suffer from position bias when serializing candidate sets into prompts. We show this order sensitivity creates an exploitable attack surface: an attacker can promote a label-0 target into the top-k solely by reordering candidates, without changing item content, labels, or model parameters. We introduce promo@k to quantify this vulnerability, measuring the fraction of label-0 targets that can be elevated into top-k rankings via permutation. Evaluating across three domains (MovieLens, Amazon Books, and Amazon Fashion), promo@5 reaches up to 0.57 at an attack budget of R = 50 orderings. Furthermore, ordinary permutation stability predicts vulnerability without running the attack. While a bidirectional T5 encoder scorer reduces exposure, permutation-consistency regularization and architectural invariance effectively mitigate it. Pointwise scoring avoids the bias issue but degrades ranking quality. These results demonstrate that input candidate order in listwise LLM reranking is a security-relevant attack vector. Code and data are available at https://github.com/geoz-lab/positionbiasattack.