# Fairkit, Fairkit, on the Wall, Who's the Fairest of Them All? Supporting Data Scientists in Training Fair Models

arXiv: https://arxiv.org/abs/2012.09951 · HTML: https://mlsanity.com/paper/2012.09951/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [Bias in Machine Learning Software: Why? How? What to do?](https://mlsanity.com/paper/2105.12195/) (2021)
- [Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits](https://mlsanity.com/paper/2205.06922/) (2022)
- [A Framework for Fairness: A Systematic Review of Existing Fair AI Solutions](https://mlsanity.com/paper/2112.05700/) (2021)
- [Fair Enough: Searching for Sufficient Measures of Fairness](https://mlsanity.com/paper/2110.13029/) (2021)
- [Generative AI voting: fair collective choice is resilient to LLM biases and inconsistencies](https://mlsanity.com/paper/2406.11871/) (2024)
- [Enhanced Fairness Testing via Generating Effective Initial Individual Discriminatory Instances](https://mlsanity.com/paper/2209.08321/) (2022)
- [Fairer Software Made Easier (using "Keys")](https://mlsanity.com/paper/2107.05088/) (2021)
- [Text-to-Image Representativity Fairness Evaluation Framework](https://mlsanity.com/paper/2410.14201/) (2024)
- [Your Model Is Unfair, Are You Even Aware? Inverse Relationship Between Comprehension and Trust in Explainability Visualizations of Biased ML Models](https://mlsanity.com/paper/2508.00140/) (2025)

## Abstract (arXiv)

Modern software relies heavily on data and machine learning, and affects decisions that shape our world. Unfortunately, recent studies have shown that because of biases in data, software systems frequently inject bias into their decisions, from producing better closed caption transcriptions of men's voices than of women's voices to overcharging people of color for financial loans. To address bias in machine learning, data scientists need tools that help them understand the trade-offs between model quality and fairness in their specific data domains. Toward that end, we present fairkit-learn, a toolkit for helping data scientists reason about and understand fairness. Fairkit-learn works with state-of-the-art machine learning tools and uses the same interfaces to ease adoption. It can evaluate thousands of models produced by multiple machine learning algorithms, hyperparameters, and data permutations, and compute and visualize a small Pareto-optimal set of models that describe the optimal trade-offs between fairness and quality. We evaluate fairkit-learn via a user study with 54 students, showing that students using fairkit-learn produce models that provide a better balance between fairness and quality than students using scikit-learn and IBM AI Fairness 360 toolkits. With fairkit-learn, users can select models that are up to 67% more fair and 10% more accurate than the models they are likely to train with scikit-learn.