# CFaiRLLM: Consumer Fairness Evaluation in Large-Language Model Recommender System

arXiv: https://arxiv.org/abs/2403.05668 · HTML: https://mlsanity.com/paper/2403.05668/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2024.

## Built on by (most cited first, 18 of 18)

- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [Poison-RAG: Adversarial Data Poisoning Attacks on Retrieval-Augmented Generation in Recommender Systems](https://mlsanity.com/paper/2501.11759/) (2025)
- [A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms](https://mlsanity.com/paper/2504.16420/) (2025)
- ["Global is Good, Local is Bad?": Understanding Brand Bias in LLMs](https://mlsanity.com/paper/2406.13997/) (2024)
- [Toward Holistic Evaluation of Recommender Systems Powered by Generative Models](https://mlsanity.com/paper/2504.06667/) (2025)
- [Exposing Product Bias in LLM Investment Recommendation](https://mlsanity.com/paper/2503.08750/) (2025)
- [A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender System](https://mlsanity.com/paper/2405.02219/) (2024)
- [Stairway to Fairness: Connecting Group and Individual Fairness](https://mlsanity.com/paper/2508.21334/) (2025)
- [PerFairX: Is There a Balance Between Fairness and Personality in Large Language Model Recommendations?](https://mlsanity.com/paper/2509.08829/) (2025)
- [Biases in Edge Language Models: Detection, Analysis, and Mitigation](https://mlsanity.com/paper/2502.11349/) (2025)
- [Augment or Not? A Comparative Study of Pure and Augmented Large Language Model Recommenders](https://mlsanity.com/paper/2505.23053/) (2025)
- [From Personalization to Prejudice: Bias and Discrimination in Memory-Enhanced AI Agents for Recruitment](https://mlsanity.com/paper/2512.16532/) (2025)
- [Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems](https://mlsanity.com/paper/2602.02582/) (2026)
- [Lightweight Fairness for LLM-Based Recommendations via Kernelized Projection and Gated Adapters](https://mlsanity.com/paper/2603.23780/) (2026)
- [Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation](https://mlsanity.com/paper/2604.15937/) (2026)
- [HALF: Harm-Aware LLM Fairness Evaluation Aligned with Deployment](https://mlsanity.com/paper/2510.12217/) (2025)
- [MedicalRec: Medical recommender system for image classification without retraining](https://mlsanity.com/paper/2606.07553/) (2026)

## Abstract (arXiv)

This work takes a critical stance on previous studies concerning fairness evaluation in Large Language Model (LLM)-based recommender systems, which have primarily assessed consumer fairness by comparing recommendation lists generated with and without sensitive user attributes. Such approaches implicitly treat discrepancies in recommended items as biases, overlooking whether these changes might stem from genuine personalization aligned with the true preferences of users. Moreover, these earlier studies typically address single sensitive attributes in isolation, neglecting the complex interplay of intersectional identities. In response to these shortcomings, we introduce CFaiRLLM, an enhanced evaluation framework that not only incorporates true preference alignment but also rigorously examines intersectional fairness by considering overlapping sensitive attributes. Additionally, CFaiRLLM introduces diverse user profile sampling strategies-random, top-rated, and recency-focused-to better understand the impact of profile generation fed to LLMs in light of inherent token limitations in these systems. Given that fairness depends on accurately understanding users' tastes and preferences, these strategies provide a more realistic assessment of fairness within RecLLMs. To validate the efficacy of CFaiRLLM, we conducted extensive experiments using MovieLens and LastFM datasets, applying various sampling strategies and sensitive attribute configurations. The evaluation metrics include both item similarity measures and true preference alignment considering both hit and ranking (Jaccard Similarity and PRAG), thereby conducting a multifaceted analysis of recommendation fairness.