# An Empirical Study on Learning Fairness Metrics for COMPAS Data with Human Supervision

arXiv: https://arxiv.org/abs/1910.10255 · HTML: https://mlsanity.com/paper/1910.10255/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2019.

## Built on by (most cited first, 13 of 13)

- [Latent Space Smoothing for Individually Fair Representations](https://mlsanity.com/paper/2111.13650/) (2021)
- [Fair Enough: Searching for Sufficient Measures of Fairness](https://mlsanity.com/paper/2110.13029/) (2021)
- [Statistical inference for individual fairness](https://mlsanity.com/paper/2103.16714/) (2021)
- [Individually Fair Gradient Boosting](https://mlsanity.com/paper/2103.16785/) (2021)
- [Individually Fair Ranking](https://mlsanity.com/paper/2103.11023/) (2021)
- [Perspectives on Incorporating Expert Feedback into Model Updates](https://mlsanity.com/paper/2205.06905/) (2022)
- [iFlipper: Label Flipping for Individual Fairness](https://mlsanity.com/paper/2209.07047/) (2022)
- [Human-Guided Fair Classification for Natural Language Processing](https://mlsanity.com/paper/2212.10154/) (2022)
- [Beyond traditional assumptions in fair machine learning](https://mlsanity.com/paper/2101.12476/) (2021)
- [A Comparative Simulation Study of the Fairness and Accuracy of Predictive Policing Systems in Baltimore City](https://mlsanity.com/paper/2602.02566/) (2026)
- [Comparing Apples to Oranges: Learning Similarity Functions for Data Produced by Different Distributions](https://mlsanity.com/paper/2208.12731/) (2022)
- [Counterpart Fairness -- Addressing Systematic between-group Differences in Fairness Evaluation](https://mlsanity.com/paper/2305.18160/) (2023)
- [Effort-aware Fairness: Incorporating a Philosophy-informed, Human-centered Notion of Effort into Algorithmic Fairness Metrics](https://mlsanity.com/paper/2505.19317/) (2025)

## Abstract (arXiv)

The notion of individual fairness requires that similar people receive similar treatment. However, this is hard to achieve in practice since it is difficult to specify the appropriate similarity metric. In this work, we attempt to learn such similarity metric from human annotated data. We gather a new dataset of human judgments on a criminal recidivism prediction (COMPAS) task. By assuming the human supervision obeys the principle of individual fairness, we leverage prior work on metric learning, evaluate the performance of several metric learning methods on our dataset, and show that the learned metrics outperform the Euclidean and Precision metric under various criteria. We do not provide a way to directly learn a similarity metric satisfying the individual fairness, but to provide an empirical study on how to derive the similarity metric from human supervisors, then future work can use this as a tool to understand human supervision.