# FairRec: Fairness Testing for Deep Recommender Systems

arXiv: https://arxiv.org/abs/2304.07030 · HTML: https://mlsanity.com/paper/2304.07030/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 58 references that are in the map.
- First posted on arXiv 14 Apr 2023.

## Builds on (most cited first, 28 of 28)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [Graph Neural Networks for Social Recommendation](https://mlsanity.com/paper/1902.07243/) (2019)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep & Cross Network for Ad Click Predictions](https://mlsanity.com/paper/1708.05123/) (2017)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Fairness in Recommendation Ranking through Pairwise Comparisons](https://mlsanity.com/paper/1903.00780/) (2019)
- [Fairness Testing: Testing Software for Discrimination](https://mlsanity.com/paper/1709.03221/) (2017)
- [Fairness in Deep Learning: A Computational Perspective](https://mlsanity.com/paper/1908.08843/) (2019)
- [A Clarification of the Nuances in the Fairness Metrics Landscape](https://mlsanity.com/paper/2106.00467/) (2021)
- [Beyond Parity: Fairness Objectives for Collaborative Filtering](https://mlsanity.com/paper/1705.08804/) (2017)
- [Automated Directed Fairness Testing](https://mlsanity.com/paper/1807.00468/) (2018)
- [User-oriented Fairness in Recommendation](https://mlsanity.com/paper/2104.10671/) (2021)
- [Fairness Testing: A Comprehensive Survey and Analysis of Trends](https://mlsanity.com/paper/2207.10223/) (2022)
- [Deconfounded Recommendation for Alleviating Bias Amplification](https://mlsanity.com/paper/2105.10648/) (2021)
- [FairRec: Fairness-aware News Recommendation with Decomposed Adversarial Learning](https://mlsanity.com/paper/2006.16742/) (2020)
- [User-centered Evaluation of Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2103.06364/) (2021)
- [Security and Privacy on Blockchain](https://mlsanity.com/paper/1903.07602/) (2019)
- [Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1904.04447/) (2019)
- [Testing Group Fairness via Optimal Transport Projections](https://mlsanity.com/paper/2106.01070/) (2021)
- [User Fairness in Recommender Systems](https://mlsanity.com/paper/1807.06349/) (2018)
- [Multi-stakeholder Recommendation and its Connection to Multi-sided Fairness](https://mlsanity.com/paper/1907.13158/) (2019)
- [Automatic Fairness Testing of Neural Classifiers through Adversarial Sampling](https://mlsanity.com/paper/2107.08176/) (2021)
- [Experiments on Generalizability of User-Oriented Fairness in Recommender Systems](https://mlsanity.com/paper/2205.08289/) (2022)
- [TESTSGD: Interpretable Testing of Neural Networks Against Subtle Group Discrimination](https://mlsanity.com/paper/2208.11321/) (2022)
- [A Pre-processing Method for Fairness in Ranking](https://mlsanity.com/paper/2110.15503/) (2021)

## Built on by (most cited first, 3 of 3)

- [Do Existing Testing Tools Really Uncover Gender Bias in Text-to-Image Models?](https://mlsanity.com/paper/2501.15775/) (2025)
- [Search-Based Fairness Testing: An Overview](https://mlsanity.com/paper/2311.06175/) (2023)
- [Fairpriori: Improving Biased Subgroup Discovery for Deep Neural Network Fairness](https://mlsanity.com/paper/2407.01595/) (2024)

## Abstract (arXiv)

Deep learning-based recommender systems (DRSs) are increasingly and widely deployed in the industry, which brings significant convenience to people's daily life in different ways. However, recommender systems are also shown to suffer from multiple issues,e.g., the echo chamber and the Matthew effect, of which the notation of "fairness" plays a core role.While many fairness notations and corresponding fairness testing approaches have been developed for traditional deep classification models, they are essentially hardly applicable to DRSs. One major difficulty is that there still lacks a systematic understanding and mapping between the existing fairness notations and the diverse testing requirements for deep recommender systems, not to mention further testing or debugging activities. To address the gap, we propose FairRec, a unified framework that supports fairness testing of DRSs from multiple customized perspectives, e.g., model utility, item diversity, item popularity, etc. We also propose a novel, efficient search-based testing approach to tackle the new challenge, i.e., double-ended discrete particle swarm optimization (DPSO) algorithm, to effectively search for hidden fairness issues in the form of certain disadvantaged groups from a vast number of candidate groups. Given the testing report, by adopting a simple re-ranking mitigation strategy on these identified disadvantaged groups, we show that the fairness of DRSs can be significantly improved. We conducted extensive experiments on multiple industry-level DRSs adopted by leading companies. The results confirm that FairRec is effective and efficient in identifying the deeply hidden fairness issues, e.g., achieving 95% testing accuracy with half to 1/8 time.