# Putting Fairness Principles into Practice: Challenges, Metrics, and Improvements

arXiv: https://arxiv.org/abs/1901.04562 · HTML: https://mlsanity.com/paper/1901.04562/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2019.

## Built on by (most cited first, 30 of 70)

- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [Addressing Algorithmic Disparity and Performance Inconsistency in Federated Learning](https://mlsanity.com/paper/2108.08435/) (2021)
- [A Step Toward More Inclusive People Annotations for Fairness](https://mlsanity.com/paper/2105.02317/) (2021)
- [A Survey on Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2308.01118/) (2023)
- [Unbiased Supervised Contrastive Learning](https://mlsanity.com/paper/2211.05568/) (2022)
- [FARF: A Fair and Adaptive Random Forests Classifier](https://mlsanity.com/paper/2108.07403/) (2021)
- [Representation Bias in Data: A Survey on Identification and Resolution Techniques](https://mlsanity.com/paper/2203.11852/) (2022)
- [Understanding and Improving Fairness-Accuracy Trade-offs in Multi-Task Learning](https://mlsanity.com/paper/2106.02705/) (2021)
- [Fairness in Recommendation: Foundations, Methods and Applications](https://mlsanity.com/paper/2205.13619/) (2022)
- [Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems](https://mlsanity.com/paper/2101.09385/) (2021)
- [A Stochastic Optimization Framework for Fair Risk Minimization](https://mlsanity.com/paper/2102.12586/) (2021)
- [Longitudinal Fairness with Censorship](https://mlsanity.com/paper/2203.16024/) (2022)
- [Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems](https://mlsanity.com/paper/2103.06172/) (2021)
- [Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development](https://mlsanity.com/paper/2110.02932/) (2021)
- [Technical Challenges for Training Fair Neural Networks](https://mlsanity.com/paper/2102.06764/) (2021)
- [A Survey on Fairness for Machine Learning on Graphs](https://mlsanity.com/paper/2205.05396/) (2022)
- [Efficient Conditionally Invariant Representation Learning](https://mlsanity.com/paper/2212.08645/) (2022)
- [Modeling Techniques for Machine Learning Fairness: A Survey](https://mlsanity.com/paper/2111.03015/) (2021)
- [FRAPPE: A Group Fairness Framework for Post-Processing Everything](https://mlsanity.com/paper/2312.02592/) (2023)
- [Fair Decision-making Under Uncertainty](https://mlsanity.com/paper/2301.12364/) (2023)
- [Towards Equity and Algorithmic Fairness in Student Grade Prediction](https://mlsanity.com/paper/2105.06604/) (2021)
- [FIFA: Making Fairness More Generalizable in Classifiers Trained on Imbalanced Data](https://mlsanity.com/paper/2206.02792/) (2022)
- [Speciesist bias in AI -- How AI applications perpetuate discrimination and unfair outcomes against animals](https://mlsanity.com/paper/2202.10848/) (2022)
- [Equality of opportunity in travel behavior prediction with deep neural networks and discrete choice models](https://mlsanity.com/paper/2109.12422/) (2021)
- [Fairness and Bias in Robot Learning](https://mlsanity.com/paper/2207.03444/) (2022)
- [Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation](https://mlsanity.com/paper/2410.10995/) (2024)
- [Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach](https://mlsanity.com/paper/2109.08549/) (2021)
- [Stochastic Methods for AUC Optimization subject to AUC-based Fairness Constraints](https://mlsanity.com/paper/2212.12603/) (2022)
- [In the Name of Fairness: Assessing the Bias in Clinical Record De-identification](https://mlsanity.com/paper/2305.11348/) (2023)

## Abstract (arXiv)

As more researchers have become aware of and passionate about algorithmic fairness, there has been an explosion in papers laying out new metrics, suggesting algorithms to address issues, and calling attention to issues in existing applications of machine learning. This research has greatly expanded our understanding of the concerns and challenges in deploying machine learning, but there has been much less work in seeing how the rubber meets the road. In this paper we provide a case-study on the application of fairness in machine learning research to a production classification system, and offer new insights in how to measure and address algorithmic fairness issues. We discuss open questions in implementing equality of opportunity and describe our fairness metric, conditional equality, that takes into account distributional differences. Further, we provide a new approach to improve on the fairness metric during model training and demonstrate its efficacy in improving performance for a real-world product