mlsanityData to 7 Oct 2026

Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders

arXiv 2607.24869 ↗First posted 26 Jul 2026cs.IR cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) used as listwise rerankers in recommendation systems suffer from position bias when serializing candidate sets into prompts. We show this order sensitivity creates an exploitable attack surface: an attacker can promote a label-0 target into the top-k solely by reordering candidates, without changing item content, labels, or model parameters. We introduce promo@k to quantify this vulnerability, measuring the fraction of label-0 targets that can be elevated into top-k rankings via permutation. Evaluating across three domains (MovieLens, Amazon Books, and Amazon Fashion), promo@5 reaches up to 0.57 at an attack budget of R = 50 orderings. Furthermore, ordinary permutation stability predicts vulnerability without running the attack. While a bidirectional T5 encoder scorer reduces exposure, permutation-consistency regularization and architectural invariance effectively mitigate it. Pointwise scoring avoids the bias issue but degrades ranking quality. These results demonstrate that input candidate order in listwise LLM reranking is a security-relevant attack vector. Code and data are available at https://github.com/geoz-lab/positionbiasattack.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 40 references that are in the map. First posted on arXiv 26 Jul 2026.

Builds on

20 of 40 references.

References · 20

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,134
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  6. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  7. 2016Unbiased Learning-to-Rank with Biased Feedbackcited by 155
  8. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 151
  9. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
  10. 2018Unbiased Learning to Rank with Unbiased Propensity Estimationcited by 61
  11. 2024ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrievalcited by 29
  12. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
8 more
  1. 2025Evaluating Position Bias in Large Language Model Recommendationscited by 14
  2. 2024Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Datasetcited by 10
  3. 2025StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimizationcited by 10
  4. 2025Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimizationcited by 8
  5. 2024Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Modelscited by 6
  6. 2026One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendationcited by 4
  7. 2025SimAug: Enhancing Recommendation with Pretrained Language Models for Dense and Balanced Data Augmentationcited by 2
  8. 2026LLM-based Listwise Reranking under the Effect of Positional Biascited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agentscited by 0