# User-centered Evaluation of Popularity Bias in Recommender Systems

arXiv: https://arxiv.org/abs/2103.06364 · HTML: https://mlsanity.com/paper/2103.06364/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 50 references that are in the map.
- First posted on arXiv 10 Mar 2021.

## Builds on (most cited first, 10 of 10)

- [FA*IR: A Fair Top-k Ranking Algorithm](https://mlsanity.com/paper/1706.06368/) (2017)
- [Beyond Parity: Fairness Objectives for Collaborative Filtering](https://mlsanity.com/paper/1705.08804/) (2017)
- [Feedback Loop and Bias Amplification in Recommender Systems](https://mlsanity.com/paper/2007.13019/) (2020)
- [Managing Popularity Bias in Recommender Systems with Personalized Re-ranking](https://mlsanity.com/paper/1901.07555/) (2019)
- [The Unfairness of Popularity Bias in Recommendation](https://mlsanity.com/paper/1907.13286/) (2019)
- [The Unfairness of Popularity Bias in Music Recommendation: A Reproducibility Study](https://mlsanity.com/paper/1912.04696/) (2019)
- [How algorithmic popularity bias hinders or promotes quality](https://mlsanity.com/paper/1707.00574/) (2017)
- [FairMatch: A Graph-based Approach for Improving Aggregate Diversity in Recommender Systems](https://mlsanity.com/paper/2005.01148/) (2020)
- [Popularity Bias in Recommendation: A Multi-stakeholder Perspective](https://mlsanity.com/paper/2008.08551/) (2020)
- [Using Stable Matching to Optimize the Balance between Accuracy and Diversity in Recommendation](https://mlsanity.com/paper/2006.03715/) (2020)

## Built on by (most cited first, 30 of 38)

- [CoRAL: Collaborative Retrieval-Augmented Large Language Models Improve Long-tail Recommendation](https://mlsanity.com/paper/2403.06447/) (2024)
- [A Survey on Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2308.01118/) (2023)
- [AI and the Problem of Knowledge Collapse](https://mlsanity.com/paper/2404.03502/) (2024)
- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Fairness in Recommender Systems: Research Landscape and Future Directions](https://mlsanity.com/paper/2205.11127/) (2022)
- [CPFair: Personalized Consumer and Producer Fairness Re-ranking for Recommender Systems](https://mlsanity.com/paper/2204.08085/) (2022)
- [The Unfairness of Popularity Bias in Book Recommendation](https://mlsanity.com/paper/2202.13446/) (2022)
- [Experiments on Generalizability of User-Oriented Fairness in Recommender Systems](https://mlsanity.com/paper/2205.08289/) (2022)
- [Do LLMs Memorize Recommendation Datasets? A Preliminary Study on MovieLens-1M](https://mlsanity.com/paper/2505.10212/) (2025)
- [The Unfairness of Active Users and Popularity Bias in Point-of-Interest Recommendation](https://mlsanity.com/paper/2202.13307/) (2022)
- [Popularity Bias in Collaborative Filtering-Based Multimedia Recommender Systems](https://mlsanity.com/paper/2203.00376/) (2022)
- [Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairness](https://mlsanity.com/paper/2409.10825/) (2024)
- [Causal Disentanglement with Network Information for Debiased Recommendations](https://mlsanity.com/paper/2204.07221/) (2022)
- [FairRec: Fairness Testing for Deep Recommender Systems](https://mlsanity.com/paper/2304.07030/) (2023)
- [Metrics for popularity bias in dynamic recommender systems](https://mlsanity.com/paper/2310.08455/) (2023)
- [A Personalized Framework for Consumer and Producer Group Fairness Optimization in Recommender Systems](https://mlsanity.com/paper/2402.00485/) (2024)
- [Performative Debias with Fair-exposure Optimization Driven by Strategic Agents in Recommender Systems](https://mlsanity.com/paper/2406.17475/) (2024)
- [Unmasking Gender Bias in Recommendation Systems and Enhancing Category-Aware Fairness](https://mlsanity.com/paper/2502.17921/) (2025)
- [Calibrated Recommendations with Contextual Bandits](https://mlsanity.com/paper/2509.05460/) (2025)
- [Biases in Scholarly Recommender Systems: Impact, Prevalence, and Mitigation](https://mlsanity.com/paper/2301.07483/) (2023)
- [Fairness Vs. Personalization: Towards Equity in Epistemic Utility](https://mlsanity.com/paper/2309.11503/) (2023)
- [Algorithmic Fairness: A Tolerance Perspective](https://mlsanity.com/paper/2405.09543/) (2024)
- [Privacy-Utility-Bias Trade-offs for Privacy-Preserving Recommender Systems](https://mlsanity.com/paper/2511.22515/) (2025)
- [Towards Selection as Power: Bounding Decision Authority in Autonomous Agents](https://mlsanity.com/paper/2602.14606/) (2026)
- [Introducing a Framework and a Decision Protocol to Calibrate Recommender Systems](https://mlsanity.com/paper/2204.03706/) (2022)
- [Multi-Objective Recommender Systems: Survey and Challenges](https://mlsanity.com/paper/2210.10309/) (2022)
- [When Collaborative Filtering is not Collaborative: Unfairness of PCA for Recommendations](https://mlsanity.com/paper/2310.09687/) (2023)
- [MOReGIn: Multi-Objective Recommendation at the Global and Individual Levels](https://mlsanity.com/paper/2401.12593/) (2024)
- [Content-Agnostic Moderation for Stance-Neutral Recommendation](https://mlsanity.com/paper/2405.18941/) (2024)
- [Investigating Language and Retrieval Bias in Multilingual Previously Fact-Checked Claim Detection](https://mlsanity.com/paper/2509.25138/) (2025)

## Abstract (arXiv)

Recommendation and ranking systems are known to suffer from popularity bias; the tendency of the algorithm to favor a few popular items while under-representing the majority of other items. Prior research has examined various approaches for mitigating popularity bias and enhancing the recommendation of long-tail, less popular, items. The effectiveness of these approaches is often assessed using different metrics to evaluate the extent to which over-concentration on popular items is reduced. However, not much attention has been given to the user-centered evaluation of this bias; how different users with different levels of interest towards popular items are affected by such algorithms. In this paper, we show the limitations of the existing metrics to evaluate popularity bias mitigation when we want to assess these algorithms from the users' perspective and we propose a new metric that can address these limitations. In addition, we present an effective approach that mitigates popularity bias from the user-centered point of view. Finally, we investigate several state-of-the-art approaches proposed in recent years to mitigate popularity bias and evaluate their performances using the existing metrics and also from the users' perspective. Our experimental results using two publicly-available datasets show that existing popularity bias mitigation techniques ignore the users' tolerance towards popular items. Our proposed user-centered method can tackle popularity bias effectively for different users while also improving the existing metrics.