# Unveiling Bias in Fairness Evaluations of Large Language Models: A Critical Literature Review of Music and Movie Recommendation Systems

arXiv: https://arxiv.org/abs/2401.04057 · HTML: https://mlsanity.com/paper/2401.04057/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 29 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Builds on (most cited first, 25 of 25)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration](https://mlsanity.com/paper/2306.09093/) (2023)
- [LLMRec: Large Language Models with Graph Augmentation for Recommendation](https://mlsanity.com/paper/2311.00423/) (2023)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations](https://mlsanity.com/paper/2308.16505/) (2023)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [FairRec: Two-Sided Fairness for Personalized Recommendations in Two-Sided Platforms](https://mlsanity.com/paper/2002.10764/) (2020)
- [Towards Long-term Fairness in Recommendation](https://mlsanity.com/paper/2101.03584/) (2021)
- [LLM-Rec: Personalized Recommendation via Prompting Large Language Models](https://mlsanity.com/paper/2307.15780/) (2023)
- [Can ChatGPT Assess Human Personalities? A General Evaluation Framework](https://mlsanity.com/paper/2303.01248/) (2023)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles](https://mlsanity.com/paper/2108.09355/) (2021)
- [UP5: Unbiased Foundation Model for Fairness-aware Recommendation](https://mlsanity.com/paper/2305.12090/) (2023)
- [Fairness in Recommendation: Foundations, Methods and Applications](https://mlsanity.com/paper/2205.13619/) (2022)
- [Evaluating Psychological Safety of Large Language Models](https://mlsanity.com/paper/2212.10529/) (2022)
- [Editing Personality for Large Language Models](https://mlsanity.com/paper/2310.02168/) (2023)
- [Fairness of ChatGPT](https://mlsanity.com/paper/2305.18569/) (2023)
- [A Preliminary Study of ChatGPT on News Recommendation: Personalization, Provider Fairness, Fake News](https://mlsanity.com/paper/2306.10702/) (2023)
- [Fairness-aware Personalized Ranking Recommendation via Adversarial Learning](https://mlsanity.com/paper/2103.07849/) (2021)
- [Modeling Artist Preferences of Users with Different Music Consumption Patterns for Fair Music Recommendations](https://mlsanity.com/paper/1907.09781/) (2019)
- [Personality Profiling: How informative are social media profiles in predicting personal information?](https://mlsanity.com/paper/2309.13065/) (2023)

## Built on by (most cited first, 7 of 7)

- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [FairEval: Evaluating Fairness in LLM-Based Recommendations with Personality Awareness](https://mlsanity.com/paper/2504.07801/) (2025)
- [Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairness](https://mlsanity.com/paper/2409.10825/) (2024)
- [Exposing Product Bias in LLM Investment Recommendation](https://mlsanity.com/paper/2503.08750/) (2025)
- [Revealing Potential Biases in LLM-Based Recommender Systems in the Cold Start Setting](https://mlsanity.com/paper/2508.20401/) (2025)
- [PerFairX: Is There a Balance Between Fairness and Personality in Large Language Model Recommendations?](https://mlsanity.com/paper/2509.08829/) (2025)
- [Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems](https://mlsanity.com/paper/2602.02582/) (2026)

## Abstract (arXiv)

The rise of generative artificial intelligence, particularly Large Language Models (LLMs), has intensified the imperative to scrutinize fairness alongside accuracy. Recent studies have begun to investigate fairness evaluations for LLMs within domains such as recommendations. Given that personalization is an intrinsic aspect of recommendation systems, its incorporation into fairness assessments is paramount. Yet, the degree to which current fairness evaluation frameworks account for personalization remains unclear. Our comprehensive literature review aims to fill this gap by examining how existing frameworks handle fairness evaluations of LLMs, with a focus on the integration of personalization factors. Despite an exhaustive collection and analysis of relevant works, we discovered that most evaluations overlook personalization, a critical facet of recommendation systems, thereby inadvertently perpetuating unfair practices. Our findings shed light on this oversight and underscore the urgent need for more nuanced fairness evaluations that acknowledge personalization. Such improvements are vital for fostering equitable development within the AI community.