# FairEval: Evaluating Fairness in LLM-Based Recommendations with Personality Awareness

arXiv: https://arxiv.org/abs/2504.07801 · HTML: https://mlsanity.com/paper/2504.07801/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 65 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Builds on (most cited first, 28 of 28)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets](https://mlsanity.com/paper/2106.10328/) (2021)
- [Fairness in Recommendation Ranking through Pairwise Comparisons](https://mlsanity.com/paper/1903.00780/) (2019)
- [Equity of Attention: Amortizing Individual Fairness in Rankings](https://mlsanity.com/paper/1805.01788/) (2018)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [Beyond Parity: Fairness Objectives for Collaborative Filtering](https://mlsanity.com/paper/1705.08804/) (2017)
- [Managing Popularity Bias in Recommender Systems with Personalized Re-ranking](https://mlsanity.com/paper/1901.07555/) (2019)
- [Multisided Fairness for Recommendation](https://mlsanity.com/paper/1707.00093/) (2017)
- [Thinking Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models](https://mlsanity.com/paper/2405.10431/) (2024)
- [Large Language Models as Recommender Systems: A Study of Popularity Bias](https://mlsanity.com/paper/2406.01285/) (2024)
- [Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Tasks](https://mlsanity.com/paper/2305.08714/) (2023)
- [Joint Multisided Exposure Fairness for Recommendation](https://mlsanity.com/paper/2205.00048/) (2022)
- [Fairness and Diversity in Recommender Systems: A Survey](https://mlsanity.com/paper/2307.04644/) (2023)
- [A Survey on Personality-Aware Recommendation Systems](https://mlsanity.com/paper/2101.12153/) (2021)
- [A Framework for Fairness: A Systematic Review of Existing Fair AI Solutions](https://mlsanity.com/paper/2112.05700/) (2021)
- [Fairness of ChatGPT and the Role Of Explainable-Guided Prompts](https://mlsanity.com/paper/2307.11761/) (2023)
- [Large Language Models are Not Stable Recommender Systems](https://mlsanity.com/paper/2312.15746/) (2023)
- [Opportunistic Multi-aspect Fairness through Personalized Re-ranking](https://mlsanity.com/paper/2005.12974/) (2020)
- [Personality Traits and Echo Chambers on Facebook](https://mlsanity.com/paper/1606.04721/) (2016)
- [Dynamic fairness-aware recommendation through multi-agent social choice](https://mlsanity.com/paper/2303.00968/) (2023)
- [ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods](https://mlsanity.com/paper/2409.16965/) (2024)
- [Unveiling Bias in Fairness Evaluations of Large Language Models: A Critical Literature Review of Music and Movie Recommendation Systems](https://mlsanity.com/paper/2401.04057/) (2024)

## Built on by (most cited first, 8 of 8)

- [Conv-FinRe: A Conversational and Longitudinal Benchmark for Utility-Grounded Financial Recommendation](https://mlsanity.com/paper/2602.16990/) (2026)
- [PerFairX: Is There a Balance Between Fairness and Personality in Large Language Model Recommendations?](https://mlsanity.com/paper/2509.08829/) (2025)
- [IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia](https://mlsanity.com/paper/2603.17915/) (2026)
- [Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interest](https://mlsanity.com/paper/2604.08525/) (2026)
- [Emergence of Biased Consensus in Multi-Agent LLM Debates](https://mlsanity.com/paper/2608.02827/) (2026)
- [Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation](https://mlsanity.com/paper/2604.15937/) (2026)
- [PBiLoss: Popularity-Aware Regularization to Improve Fairness in Graph-Based Recommender Systems](https://mlsanity.com/paper/2507.19067/) (2025)
- [Rethinking Fairness in LLM-Based Recommender Systems: A Survey](https://mlsanity.com/paper/2606.28340/) (2026)

## Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have enabled their application to recommender systems (RecLLMs), yet concerns remain regarding fairness across demographic and psychological user dimensions. We introduce FairEval, a novel evaluation framework to systematically assess fairness in LLM-based recommendations. FairEval integrates personality traits with eight sensitive demographic attributes,including gender, race, and age, enabling a comprehensive assessment of user-level bias. We evaluate models, including ChatGPT 4o and Gemini 1.5 Flash, on music and movie recommendations. FairEval's fairness metric, PAFS, achieves scores up to 0.9969 for ChatGPT 4o and 0.9997 for Gemini 1.5 Flash, with disparities reaching 34.79 percent. These results highlight the importance of robustness in prompt sensitivity and support more inclusive recommendation systems.