mlsanityData to 7 Oct 2026

Fair ranking: a critical review, challenges, and future directions

arXiv 2201.12662 ↗First posted 29 Jan 2022cs.IR cs.AI cs.GTcited by 21

Not on the map. Why?

Abstract (arXiv)

Ranking, recommendation, and retrieval systems are widely used in online platforms and other societal systems, including e-commerce, media-streaming, admissions, gig platforms, and hiring. In the recent past, a large "fair ranking" research literature has been developed around making these systems fair to the individuals, providers, or content that are being ranked. Most of this literature defines fairness for a single instance of retrieval, or as a simple additive notion for multiple instances of retrievals over time. This work provides a critical overview of this literature, detailing the often context-specific concerns that such an approach misses: the gap between high ranking placements and true provider utility, spillovers and compounding effects over time, induced strategic incentives, and the effect of statistical uncertainty. We then provide a path forward for a more holistic and impact-oriented fair ranking research agenda, including methodological lessons from other fields and the role of the broader stakeholder community in overcoming data bottlenecks and designing effective regulatory environments.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 69 of its 189 references that are in the map. First posted on arXiv 29 Jan 2022.

Builds on

69 of 189 references.

References · 69

  1. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  2. 2018Datasheets for Datasetscited by 1,371
  3. 2020Fairness without Demographics through Adversarially Reweighted Learningcited by 251
  4. 2016Recommendations as Treatments: Debiasing Learning and Evaluationcited by 246
  5. 2020Bias and Debias in Recommender System: A Survey and Future Directionscited by 239
  6. 2020A Survey on Causal Inferencecited by 202
  7. 2018Fairness of Exposure in Rankingscited by 189
  8. 2018Delayed Impact of Fair Machine Learningcited by 183
  9. 2016Unbiased Learning-to-Rank with Biased Feedbackcited by 155
  10. 2019Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Searchcited by 137
  11. 2018Equity of Attention: Amortizing Individual Fairness in Rankingscited by 124
  12. 2017How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utilitycited by 116
38 more
  1. 2017FA*IR: A Fair Top-k Ranking Algorithmcited by 113
  2. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  3. 2018A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indicescited by 97
  4. 2017Beyond Parity: Fairness Objectives for Collaborative Filteringcited by 93
  5. 2020Feedback Loop and Bias Amplification in Recommender Systemscited by 89
  6. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  7. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  8. 2016Data Poisoning Attacks on Factorization-Based Collaborative Filteringcited by 78
  9. 2017Ranking with Fairness Constraintscited by 78
  10. 2020FairRec: Two-Sided Fairness for Personalized Recommendations in Two-Sided Platformscited by 77
  11. 2021Towards Long-term Fairness in Recommendationcited by 77
  12. 2017Multisided Fairness for Recommendationcited by 69
  13. 2020Controlling Fairness and Bias in Dynamic Learning-to-Rankcited by 67
  14. 2018RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertisingcited by 65
  15. 2020"What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairnesscited by 56
  16. 2020Evaluating Stochastic Rankings with Expected Exposurecited by 53
  17. 2017Designing Fair Ranking Schemescited by 40
  18. 2015COEVOLVE: A Joint Point Process Model for Information Diffusion and Network Co-evolutioncited by 37
  19. 2018Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Applicationcited by 33
  20. 2018A Game-Theoretic Approach to Recommendation Systems with Strategic Content Providerscited by 33
  21. 2021Fairness in Ranking: A Surveycited by 33
  22. 2020Optimizing Long-term Social Welfare in Recommender Systems: A Constrained Matching Approachcited by 32
  23. 2018Estimating Position Bias without Intrusive Interventionscited by 31
  24. 2020Bridging Machine Learning and Mechanism Design towards Algorithmic Fairnesscited by 30
  25. 2021Two-sided fairness in rankings via Lorenz dominancecited by 28
  26. 2020Practical Data Poisoning Attack against Next-Item Recommendationcited by 27
  27. 2020Do Offline Metrics Predict Online Performance in Recommender Systems?cited by 25
  28. 2019Quantifying the Impact of User Attention on Fair Group Representation in Ranked Listscited by 24
  29. 2019Awareness in Practice: Tensions in Access to Sensitive Attribute Data for Antidiscriminationcited by 23
  30. 2019FairSearch: A Tool For Fairness in Ranked Search Resultscited by 22
  31. 2021When Fair Ranking Meets Uncertain Inferencecited by 22
  32. 2021Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studiescited by 22
  33. 2015Estimating the Causal Impact of Recommendation Systems from Observational Datacited by 21
  34. 2021RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystemscited by 21
  35. 2021Strategic Rankingcited by 21
  36. 2021Measuring Recommender System Effects with Simulated Userscited by 20
  37. 2020Operationalizing the Legal Principle of Data Minimization for Personalizationcited by 19
  38. 2021Maxmin-Fair Ranking: Individual Fairness under Group-Fairness Constraintscited by 18

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 7; 2024: 5; 2025: 4.

21 papers.

Sort:

By area:

  1. 2022Supply-Side Equilibria in Recommender Systemscited by 33
  2. 2022Fairness in Recommendation: Foundations, Methods and Applicationscited by 27
  3. 2022Fair Ranking with Noisy Protected Attributescited by 17
  4. 2023A Review of the Role of Causality in Developing Trustworthy AI Systemscited by 14
  5. 2022Fair Ranking as Fair Division: Impact-Based Individual Fairness in Rankingcited by 13
  6. 2023Dynamic fairness-aware recommendation through multi-agent social choicecited by 9
  7. 2023Fairness in Matching under Uncertaintycited by 5
  8. 2023Maximizing Submodular Functions for Recommendation in the Presence of Biasescited by 5
  9. 2024Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challengescited by 4
  10. 2023The Role of Relevance in Fair Rankingcited by 3
  11. 2022Contextual bandits with concave rewards, and an application to fair rankingcited by 2
  12. 2023Subset Selection Based On Multiple Rankings in the Presence of Bias: Effectiveness of Fairness Constraints for Multiwinner Voting Score Functionscited by 2
9 more
  1. 2023Sampling Individually-Fair Rankings that are Always Group Faircited by 2
  2. 2024Hidden or Inferred: Fair Learning-To-Rank with Unknown Demographicscited by 2
  3. 2024Fair Clustering: Critique, Caveats, and Future Directionscited by 1
  4. 2024A Unified Causal Framework for Auditing Recommender Systems for Ethical Concernscited by 1
  5. 2025What's in a Query: Polarity-Aware Distribution-Based Fair Rankingcited by 1
  6. 2025From Personalization to Prejudice: Bias and Discrimination in Memory-Enhanced AI Agents for Recruitmentcited by 1
  7. 2024Social Choice for Heterogeneous Fairness in Recommendationcited by 0
  8. 2025Fairness-in-the-Workflow: How Machine Learning Practitioners at Big Tech Companies Approach Fairness in Recommender Systemscited by 0
  9. 2025Bias vs Bias -- Dawn of Justice: A Fair Fight in Recommendation Systemscited by 0