# Large Language Models as Recommender Systems: A Study of Popularity Bias

arXiv: https://arxiv.org/abs/2406.01285 · HTML: https://mlsanity.com/paper/2406.01285/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 30 of 30)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://mlsanity.com/paper/2110.08527/) (2021)
- [The Unfairness of Popularity Bias in Recommendation](https://mlsanity.com/paper/1907.13286/) (2019)
- [How Do Fair Decisions Fare in Long-term Qualification?](https://mlsanity.com/paper/2010.11300/) (2020)
- [A Survey on Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2308.01118/) (2023)
- [RecRanker: Instruction Tuning Large Language Model as Ranker for Top-k Recommendation](https://mlsanity.com/paper/2312.16018/) (2023)
- [The Connection Between Popularity Bias, Calibration, and Fairness in Recommendation](https://mlsanity.com/paper/2008.09273/) (2020)
- [Downstream Effects of Affirmative Action](https://mlsanity.com/paper/1808.09004/) (2018)
- [Popularity Bias Is Not Always Evil: Disentangling Benign and Harmful Bias for Recommendation](https://mlsanity.com/paper/2109.07946/) (2021)
- [Tapping the Potential of Large Language Models as Recommender Systems: A Comprehensive Framework and Empirical Analysis](https://mlsanity.com/paper/2401.04997/) (2024)
- [Geographical Erasure in Language Generation](https://mlsanity.com/paper/2310.14777/) (2023)
- [LensKit for Python: Next-Generation Software for Recommender System Experiments](https://mlsanity.com/paper/1809.03125/) (2018)
- [The Long Arc of Fairness: Formalisations and Ethical Discourse](https://mlsanity.com/paper/2203.06038/) (2022)
- [Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking](https://mlsanity.com/paper/2206.07247/) (2022)
- [Analyzing Item Popularity Bias of Music Recommender Systems: Are Different Genders Equally Affected?](https://mlsanity.com/paper/2108.06973/) (2021)
- [Gini estimation under infinite variance](https://mlsanity.com/paper/1707.01370/) (2017)

## Built on by (most cited first, 26 of 26)

- [CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation](https://mlsanity.com/paper/2506.21805/) (2025)
- [AgentCF++: Memory-enhanced LLM-based Agents for Popularity-aware Cross-domain Recommendations](https://mlsanity.com/paper/2502.13843/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms](https://mlsanity.com/paper/2504.16420/) (2025)
- [The Future is Agentic: Definitions, Perspectives, and Open Challenges of Multi-Agent Recommender Systems](https://mlsanity.com/paper/2507.02097/) (2025)
- [Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness](https://mlsanity.com/paper/2502.09637/) (2025)
- [FairEval: Evaluating Fairness in LLM-Based Recommendations with Personality Awareness](https://mlsanity.com/paper/2504.07801/) (2025)
- [SynthTRIPs: A Knowledge-Grounded Framework for Benchmark Query Generation for Personalized Tourism Recommenders](https://mlsanity.com/paper/2504.09277/) (2025)
- [To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay](https://mlsanity.com/paper/2505.04209/) (2025)
- [Autonomous Information Seeking: A Roadmap for Agentic Recommender Systems](https://mlsanity.com/paper/2607.04433/) (2026)
- [Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems](https://mlsanity.com/paper/2606.17443/) (2026)
- [Neutralizing Popularity Bias in LLM-based Recommendation via Counterfactual Reasoning Guidelines](https://mlsanity.com/paper/2503.08051/) (2025)
- [LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBay](https://mlsanity.com/paper/2508.03628/) (2025)
- [DenseRec: Revisiting Dense Content Embeddings for Sequential Transformer-based Recommendation](https://mlsanity.com/paper/2508.18442/) (2025)
- [Gender and Race Bias in Consumer Product Recommendations by Large Language Models](https://mlsanity.com/paper/2602.08124/) (2026)
- [Prominence-Stratified Failure Modes in Retrieval-Augmented Commercial Recommendation: A 37,000-Run Audit](https://mlsanity.com/paper/2605.27439/) (2026)
- [Persona Conditioning of Brand Recommendations in Retrieval-Augmented Commercial Chat: A Prominence-Stratified Cross-Provider Audit](https://mlsanity.com/paper/2605.30207/) (2026)
- [The Language of the Question Selects the Market: Query Language and Exit IP as Separable Factors in Commercial Recommendations from a Generative Search Interface](https://mlsanity.com/paper/2608.30052/) (2026)
- [A Language-Driven Framework for Improving Personalized Recommendations: Merging LLMs with Traditional Algorithms](https://mlsanity.com/paper/2507.07251/) (2025)
- [PBiLoss: Popularity-Aware Regularization to Improve Fairness in Graph-Based Recommender Systems](https://mlsanity.com/paper/2507.19067/) (2025)
- [Pro-AI Bias in Large Language Models](https://mlsanity.com/paper/2601.13749/) (2026)
- [Aligning Recommendations with User Popularity Preferences](https://mlsanity.com/paper/2604.01036/) (2026)
- [An Investigation of Linguistic Biases in LLM-Based Recommendations](https://mlsanity.com/paper/2604.25456/) (2026)
- [Authority Bias in Conversational Search Engines for Academic Paper Recommendation](https://mlsanity.com/paper/2609.00248/) (2026)
- ["If I Had to Buy Just ONE: Galaxy S26 Ultra": Auditing AI-Generated Product Recommendations](https://mlsanity.com/paper/2609.18729/) (2026)
- [A Systematic Investigation of Bias in Large Language Models for Advertising Relevance](https://mlsanity.com/paper/2610.07544/) (2026)

## Abstract (arXiv)

The issue of popularity bias -- where popular items are disproportionately recommended, overshadowing less popular but potentially relevant items -- remains a significant challenge in recommender systems. Recent advancements have seen the integration of general-purpose Large Language Models (LLMs) into the architecture of such systems. This integration raises concerns that it might exacerbate popularity bias, given that the LLM's training data is likely dominated by popular items. However, it simultaneously presents a novel opportunity to address the bias via prompt tuning. Our study explores this dichotomy, examining whether LLMs contribute to or can alleviate popularity bias in recommender systems. We introduce a principled way to measure popularity bias by discussing existing metrics and proposing a novel metric that fulfills a series of desiderata. Based on our new metric, we compare a simple LLM-based recommender to traditional recommender systems on a movie recommendation task. We find that the LLM recommender exhibits less popularity bias, even without any explicit mitigation.