# Evaluating Position Bias in Large Language Model Recommendations

arXiv: https://arxiv.org/abs/2508.02020 · HTML: https://mlsanity.com/paper/2508.02020/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2025.

## Built on by (most cited first, 15 of 15)

- [PathWise: Planning through World Model for Automated Heuristic Design via Self-Evolving LLMs](https://mlsanity.com/paper/2601.20539/) (2026)
- [One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation](https://mlsanity.com/paper/2604.27599/) (2026)
- [ReThinker: Scientific Reasoning by Rethinking with Guided Reflection and Confidence Control](https://mlsanity.com/paper/2602.04496/) (2026)
- [Is Your LLM-as-a-Recommender Agent Trustable? LLMs' Recommendation is Easily Hacked by Biases (Preferences)](https://mlsanity.com/paper/2603.17417/) (2026)
- [One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models](https://mlsanity.com/paper/2603.03291/) (2026)
- [AesRM: Improving Video Aesthetics with Expert-Level Feedback](https://mlsanity.com/paper/2604.28078/) (2026)
- [Gender Bias in LLMs: Preliminary Evidence from Shared Parenting Scenario in Czech Family Law](https://mlsanity.com/paper/2601.05879/) (2026)
- [Reasoning to Rank: An End-to-End Solution for Exploiting Large Language Models for Recommendation](https://mlsanity.com/paper/2602.12530/) (2026)
- [Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs](https://mlsanity.com/paper/2605.12242/) (2026)
- [Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection](https://mlsanity.com/paper/2606.16344/) (2026)
- [Ranked by Position: Order Sensitivity as an Exploitable Attack Surface in LLM Listwise Recommenders](https://mlsanity.com/paper/2607.24869/) (2026)
- [Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation](https://mlsanity.com/paper/2602.07298/) (2026)
- [FineREX: Fine-Tuned NER-RE for Human Smuggling Knowledge Graphs](https://mlsanity.com/paper/2606.19710/) (2026)
- [LBR: Towards Mitigating Length Bias in Large Language Models for Recommendation](https://mlsanity.com/paper/2607.04270/) (2026)
- [SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agents](https://mlsanity.com/paper/2609.35889/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are being increasingly explored as general-purpose tools for recommendation tasks, enabling zero-shot and instruction-following capabilities without the need for task-specific training. While the research community is enthusiastically embracing LLMs, there are important caveats to directly adapting them for recommendation tasks. In this paper, we show that LLM-based recommendation models suffer from position bias, where the order of candidate items in a prompt can disproportionately influence the recommendations produced by LLMs. First, we analyse the position bias of LLM-based recommendations on real-world datasets, where results uncover systemic biases of LLMs with high sensitivity to input orders. Furthermore, we introduce a new prompting strategy to mitigate the position bias of LLM recommendation models called Ranking via Iterative SElection (RISE). We compare our proposed method against various baselines on key benchmark datasets. Experiment results show that our method reduces sensitivity to input ordering and improves stability without requiring model fine-tuning or post-processing.