# Large Language Models are Not Stable Recommender Systems

arXiv: https://arxiv.org/abs/2312.15746 · HTML: https://mlsanity.com/paper/2312.15746/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 25 Dec 2023.

## Builds on (most cited first, 22 of 22)

- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System](https://mlsanity.com/paper/2303.14524/) (2023)
- [Is ChatGPT a Good Recommender? A Preliminary Study](https://mlsanity.com/paper/2304.10149/) (2023)
- [Towards Universal Sequence Representation Learning for Recommender Systems](https://mlsanity.com/paper/2206.05941/) (2022)
- [Generative Recommendation: Towards Next-generation Recommender Paradigm](https://mlsanity.com/paper/2304.03516/) (2023)
- [SimpleX: A Simple and Strong Baseline for Collaborative Filtering](https://mlsanity.com/paper/2109.12613/) (2021)
- [Deep CTR Prediction in Display Advertising](https://mlsanity.com/paper/1609.06018/) (2016)
- [FeedRec: News Feed Recommendation with Various User Feedbacks](https://mlsanity.com/paper/2102.04903/) (2021)
- [Sparks of Artificial General Recommender (AGR): Early Experiments with ChatGPT](https://mlsanity.com/paper/2305.04518/) (2023)

## Built on by (most cited first, 13 of 13)

- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [CoRAL: Collaborative Retrieval-Augmented Large Language Models Improve Long-tail Recommendation](https://mlsanity.com/paper/2403.06447/) (2024)
- [A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval](https://mlsanity.com/paper/2503.05659/) (2025)
- [DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph](https://mlsanity.com/paper/2406.17271/) (2024)
- [Cognitive Biases in Large Language Models: A Survey and Mitigation Experiments](https://mlsanity.com/paper/2412.00323/) (2024)
- [FairEval: Evaluating Fairness in LLM-Based Recommendations with Personality Awareness](https://mlsanity.com/paper/2504.07801/) (2025)
- [TEARS: Textual Representations for Scrutable Recommendations](https://mlsanity.com/paper/2410.19302/) (2024)
- [One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation](https://mlsanity.com/paper/2604.27599/) (2026)
- [Augment or Not? A Comparative Study of Pure and Augmented Large Language Model Recommenders](https://mlsanity.com/paper/2505.23053/) (2025)
- [Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems](https://mlsanity.com/paper/2602.02582/) (2026)
- [LlamaRec-LKG-RAG: A Single-Pass, Learnable Knowledge Graph-RAG Framework for LLM-Based Ranking](https://mlsanity.com/paper/2506.07449/) (2025)
- [SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models](https://mlsanity.com/paper/2604.09553/) (2026)
- [LBR: Towards Mitigating Length Bias in Large Language Models for Recommendation](https://mlsanity.com/paper/2607.04270/) (2026)

## Abstract (arXiv)

With the significant successes of large language models (LLMs) in many natural language processing tasks, there is growing interest among researchers in exploring LLMs for novel recommender systems. However, we have observed that directly using LLMs as a recommender system is usually unstable due to its inherent position bias. To this end, we introduce exploratory research and find consistent patterns of positional bias in LLMs that influence the performance of recommendation across a range of scenarios. Then, we propose a Bayesian probabilistic framework, STELLA (Stable LLM for Recommendation), which involves a two-stage pipeline. During the first probing stage, we identify patterns in a transition matrix using a probing detection dataset. And in the second recommendation stage, a Bayesian strategy is employed to adjust the biased output of LLMs with an entropy indicator. Therefore, our framework can capitalize on existing pattern information to calibrate instability of LLMs, and enhance recommendation performance. Finally, extensive experiments clearly validate the effectiveness of our framework.