# Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness

arXiv: https://arxiv.org/abs/2110.08353 · HTML: https://mlsanity.com/paper/2110.08353/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 44 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Fairness of Exposure in Rankings](https://mlsanity.com/paper/1802.07281/) (2018)
- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination](https://mlsanity.com/paper/1408.6491/) (2014)
- [FairRec: Two-Sided Fairness for Personalized Recommendations in Two-Sided Platforms](https://mlsanity.com/paper/2002.10764/) (2020)
- [Multisided Fairness for Recommendation](https://mlsanity.com/paper/1707.00093/) (2017)
- [Evaluating Stochastic Rankings with Expected Exposure](https://mlsanity.com/paper/2004.13157/) (2020)
- [The Connection Between Popularity Bias, Calibration, and Fairness in Recommendation](https://mlsanity.com/paper/2008.09273/) (2020)
- [Exploring Author Gender in Book Rating and Recommendation](https://mlsanity.com/paper/1808.07586/) (2018)
- [Multi-sided Exposure Bias in Recommendation](https://mlsanity.com/paper/2006.15772/) (2020)
- [Multi-stakeholder Recommendation and its Connection to Multi-sided Fairness](https://mlsanity.com/paper/1907.13158/) (2019)
- [A Study of Position Bias in Digital Library Recommender Systems](https://mlsanity.com/paper/1802.06565/) (2018)
- [Auditing Search Engines for Differential Satisfaction Across Demographics](https://mlsanity.com/paper/1705.10689/) (2017)
- [Algorithms are not neutral: Bias in collaborative filtering](https://mlsanity.com/paper/2105.01031/) (2021)

## Built on by (most cited first, 12 of 12)

- [A Survey on Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2308.01118/) (2023)
- [Joint Multisided Exposure Fairness for Recommendation](https://mlsanity.com/paper/2205.00048/) (2022)
- [General Intelligence Requires Rethinking Exploration](https://mlsanity.com/paper/2211.07819/) (2022)
- [Experiments on Generalizability of User-Oriented Fairness in Recommender Systems](https://mlsanity.com/paper/2205.08289/) (2022)
- [Personalized Reward Learning with Interaction-Grounded Learning (IGL)](https://mlsanity.com/paper/2211.15823/) (2022)
- [Unveiling and Mitigating Bias in Large Language Model Recommendations: A Path to Fairness](https://mlsanity.com/paper/2409.10825/) (2024)
- [Ethical and Social Considerations in Automatic Expert Identification and People Recommendation in Organizational Knowledge Management Systems](https://mlsanity.com/paper/2209.03819/) (2022)
- [A Comparative Analysis of Bias Amplification in Graph Neural Network Approaches for Recommender Systems](https://mlsanity.com/paper/2301.07639/) (2023)
- [Fuck the Algorithm: Conceptual Issues in Algorithmic Bias](https://mlsanity.com/paper/2505.13509/) (2025)
- [Bias vs Bias -- Dawn of Justice: A Fair Fight in Recommendation Systems](https://mlsanity.com/paper/2506.18327/) (2025)
- [MaRCA: Multi-Agent Reinforcement Learning for Dynamic Computation Allocation in Large-Scale Recommender Systems](https://mlsanity.com/paper/2512.24325/) (2025)
- [Interaction-Grounded Learning for Contextual Markov Decision Processes with Personalized Feedback](https://mlsanity.com/paper/2602.08307/) (2026)

## Abstract (arXiv)

Recommendation algorithms are susceptible to popularity bias: a tendency to recommend popular items even when they fail to meet user needs. A related issue is that the recommendation quality can vary by demographic groups. Marginalized groups or groups that are under-represented in the training data may receive less relevant recommendations from these algorithms compared to others. In a recent study, Ekstrand et al. investigate how recommender performance varies according to popularity and demographics, and find statistically significant differences in recommendation utility between binary genders on two datasets, and significant effects based on age on one dataset. Here we reproduce those results and extend them with additional analyses. We find statistically significant differences in recommender performance by both age and gender. We observe that recommendation utility steadily degrades for older users, and is lower for women than men. We also find that the utility is higher for users from countries with more representation in the dataset. In addition, we find that total usage and the popularity of consumed content are strong predictors of recommender performance and also vary significantly across demographic groups.