# Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance

arXiv: https://arxiv.org/abs/2405.18276 · HTML: https://mlsanity.com/paper/2405.18276/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2024.

## Built on by (most cited first, 4 of 4)

- [Stairway to Fairness: Connecting Group and Individual Fairness](https://mlsanity.com/paper/2508.21334/) (2025)
- [We're Still Doing It (All) Wrong: Recommender Systems, Fifteen Years Later](https://mlsanity.com/paper/2509.09414/) (2025)
- [Advanced Acceptance Score: A Holistic Measure for Biometric Quantification](https://mlsanity.com/paper/2602.15535/) (2026)
- [Exploring How Fair Model Representations Relate to Fair Recommendations](https://mlsanity.com/paper/2603.24396/) (2026)

## Abstract (arXiv)

Relevance and fairness are two major objectives of recommender systems (RSs). Recent work proposes measures of RS fairness that are either independent from relevance (fairness-only) or conditioned on relevance (joint measures). While fairness-only measures have been studied extensively, we look into whether joint measures can be trusted. We collect all joint evaluation measures of RS relevance and fairness, and ask: How much do they agree with each other? To what extent do they agree with relevance/fairness measures? How sensitive are they to changes in rank position, or to increasingly fair and relevant recommendations? We empirically study for the first time the behaviour of these measures across 4 real-world datasets and 4 recommenders. We find that most of these measures: i) correlate weakly with one another and even contradict each other at times; ii) are less sensitive to rank position changes than relevance- and fairness-only measures, meaning that they are less granular than traditional RS measures; and iii) tend to compress scores at the low end of their range, meaning that they are not very expressive. We counter the above limitations with a set of guidelines on the appropriate usage of such measures, i.e., they should be used with caution due to their tendency to contradict each other and of having a very small empirical range.