# A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender System

arXiv: https://arxiv.org/abs/2405.02219 · HTML: https://mlsanity.com/paper/2405.02219/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 9 of 9)

- [Large Language Models as Zero-Shot Conversational Recommenders](https://mlsanity.com/paper/2308.10053/) (2023)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [User-oriented Fairness in Recommendation](https://mlsanity.com/paper/2104.10671/) (2021)
- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [Fairness in Recommender Systems: Research Landscape and Future Directions](https://mlsanity.com/paper/2205.11127/) (2022)
- [Fairness in Information Access Systems](https://mlsanity.com/paper/2105.05779/) (2021)
- [Understanding Biases in ChatGPT-based Recommender Systems: Provider Fairness, Temporal Stability, and Recency](https://mlsanity.com/paper/2401.10545/) (2024)
- [CFaiRLLM: Consumer Fairness Evaluation in Large-Language Model Recommender System](https://mlsanity.com/paper/2403.05668/) (2024)
- [Consumer Fairness in Recommender Systems: Contextualizing Definitions and Mitigations](https://mlsanity.com/paper/2201.08614/) (2022)

## Built on by (most cited first, 4 of 4)

- [Stairway to Fairness: Connecting Group and Individual Fairness](https://mlsanity.com/paper/2508.21334/) (2025)
- [XAI4LLM. Let Machine Learning Models and LLMs Collaborate for Enhanced In-Context Learning in Healthcare](https://mlsanity.com/paper/2405.06270/) (2024)
- [SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models](https://mlsanity.com/paper/2604.09553/) (2026)
- [Rethinking Fairness in LLM-Based Recommender Systems: A Survey](https://mlsanity.com/paper/2606.28340/) (2026)

## Abstract (arXiv)

The rapid adoption of large language models (LLMs) in recommender systems (RS) presents new challenges in understanding and evaluating their biases, which can result in unfairness or the amplification of stereotypes. Traditional fairness evaluations in RS primarily focus on collaborative filtering (CF) settings, which may not fully capture the complexities of LLMs, as these models often inherit biases from large, unregulated data. This paper proposes a normative framework to benchmark consumer fairness in LLM-powered recommender systems (RecLLMs). We critically examine how fairness norms in classical RS fall short in addressing the challenges posed by LLMs. We argue that this gap can lead to arbitrary conclusions about fairness, and we propose a more structured, formal approach to evaluate fairness in such systems. Our experiments on the MovieLens dataset on consumer fairness, using in-context learning (zero-shot vs. few-shot) reveal fairness deviations in age-based recommendations, particularly when additional contextual examples are introduced (ICL-2). Statistical significance tests confirm that these deviations are not random, highlighting the need for robust evaluation methods. While this work offers a preliminary discussion on a proposed normative framework, our hope is that it could provide a formal, principled approach for auditing and mitigating bias in RecLLMs. The code and dataset used for this work will be shared at "gihub-anonymized".