mlsanityData to 14 Oct 2026

Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation

arXiv 2305.07609 ↗First posted 12 May 2023cs.IR cs.CL cs.CYcited by 114

Not on the map. Why?

Abstract (arXiv)

The remarkable achievements of Large Language Models (LLMs) have led to the emergence of a novel recommendation paradigm -- Recommendation via LLM (RecLLM). Nevertheless, it is important to note that LLMs may contain social prejudices, and therefore, the fairness of recommendations made by RecLLM requires further investigation. To avoid the potential risks of RecLLM, it is imperative to evaluate the fairness of RecLLM with respect to various sensitive attributes on the user side. Due to the differences between the RecLLM paradigm and the traditional recommendation paradigm, it is problematic to directly use the fairness benchmark of traditional recommendation. To address the dilemma, we propose a novel benchmark called Fairness of Recommendation via LLM (FaiRLLM). This benchmark comprises carefully crafted metrics and a dataset that accounts for eight sensitive attributes1 in two recommendation scenarios: music and movies. By utilizing our FaiRLLM benchmark, we conducted an evaluation of ChatGPT and discovered that it still exhibits unfairness to some sensitive attributes when generating recommendations. Our code and dataset can be found at https://github.com/jizhi-zhang/FaiRLLM.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 12 May 2023.

Builds on

26 of 42 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  10. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  11. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 337
  12. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 242
14 more
  1. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  2. 2019Fairness in Recommendation Ranking through Pairwise Comparisonscited by 127
  3. 2018Equity of Attention: Amortizing Individual Fairness in Rankingscited by 124
  4. 2023Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisitedcited by 124
  5. 2022A Survey on the Fairness of Recommender Systemscited by 102
  6. 2021User-oriented Fairness in Recommendationcited by 75
  7. 2019The Unfairness of Popularity Bias in Recommendationcited by 70
  8. 2023Generative Recommendation: Towards Next-generation Recommender Paradigmcited by 70
  9. 2023ChatGPT: A Meta-Analysis after 2.5 Monthscited by 51
  10. 2021Personalized Counterfactual Fairness in Recommendationcited by 38
  11. 2022Toward Pareto Efficient Fairness-Utility Trade-off inRecommendation through Reinforcement Learningcited by 29
  12. 2022An Audit of Misinformation Filter Bubbles on YouTube: Bubble Bursting and Recent Behavior Changescited by 15
  13. 2022Experiments on Generalizability of User-Oriented Fairness in Recommender Systemscited by 13
  14. 2021Leave No User Behind: Towards Improving the Utility of Recommender Systems for Non-mainstream Userscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 24; 2024: 51; 2025: 29; 2026: 10.

114 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  3. 2023A Survey on Large Language Models for Recommendationcited by 379
  4. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  5. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  6. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 142
  7. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  8. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
  9. 2023ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendationcited by 77
  10. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
  11. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  12. 2023Large Language Models for Generative Recommendation: A Survey and Visionary Discussionscited by 73
38 more
  1. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  2. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  3. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 46
  4. 2023UP5: Unbiased Foundation Model for Fairness-aware Recommendationcited by 38
  5. 2023GATGPT: A Pre-trained Large Language Model with Graph Attention Network for Spatiotemporal Imputationcited by 35
  6. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  7. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 34
  8. 2024How Can Large Language Models Understand Spatial-Temporal Data?cited by 33
  9. 2025Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learningcited by 33
  10. 2023A Large Language Model Enhanced Conversational Recommender Systemcited by 31
  11. 2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 30
  12. 2023Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Communitycited by 29
  13. 2023Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answeringcited by 29
  14. 2024A Survey of Useful LLM Evaluationcited by 29
  15. 2024Understanding Biases in ChatGPT-based Recommender Systems: Provider Fairness, Temporal Stability, and Recencycited by 28
  16. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  17. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 27
  18. 2023Is ChatGPT a Good Personality Recognizer? A Preliminary Studycited by 26
  19. 2024Can Small Language Models be Good Reasoners for Sequential Recommendation?cited by 26
  20. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  21. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  22. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  23. 2024COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Actcited by 24
  24. 2023Fairness-Aware Graph Neural Networks: A Surveycited by 22
  25. 2024FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Modelscited by 22
  26. 2024The Impossibility of Fair LLMscited by 22
  27. 2024LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Applicationcited by 21
  28. 2024BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMscited by 21
  29. 2025Towards Next-Generation Recommender Systems: A Benchmark for Personalized Recommendation Assistant with LLMscited by 21
  30. 2023Evaluating ChatGPT as a Recommender System: A Rigorous Approachcited by 20
  31. 2023Bias and Fairness in Chatbots: An Overviewcited by 20
  32. 2024Prospect Personalized Recommendation on Large Language Model-based Agent Platformcited by 19
  33. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  34. 2024Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classificationcited by 18
  35. 2024SPRec: Self-Play to Debias LLM-based Recommendationcited by 18
  36. 2024Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankerscited by 17
  37. 2024Lifelong Personalized Low-Rank Adaptation of Large Language Models for Recommendationcited by 17
  38. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17

50 most cited of 114 All 114 (JSON)