# From Soft Classifiers to Hard Decisions: How fair can we be?

arXiv: https://arxiv.org/abs/1810.02003 · HTML: https://mlsanity.com/paper/1810.02003/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2018.

## Built on by (most cited first, 16 of 16)

- [Fair Infinitesimal Jackknife: Mitigating the Influence of Biased Training Data Points Without Refitting](https://mlsanity.com/paper/2212.06803/) (2022)
- [Privacy and Fairness in Federated Learning: on the Perspective of Trade-off](https://mlsanity.com/paper/2306.14123/) (2023)
- [A Sociotechnical View of Algorithmic Fairness](https://mlsanity.com/paper/2110.09253/) (2021)
- [Enforcing Group Fairness in Algorithmic Decision Making: Utility Maximization Under Sufficiency](https://mlsanity.com/paper/2206.02237/) (2022)
- [FairCal: Fairness Calibration for Face Verification](https://mlsanity.com/paper/2106.03761/) (2021)
- [On the Fairness of Machine-Assisted Human Decisions](https://mlsanity.com/paper/2110.15310/) (2021)
- [ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods](https://mlsanity.com/paper/2409.16965/) (2024)
- [Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools](https://mlsanity.com/paper/2309.17337/) (2023)
- [Algorithmic Fairness in Performative Policy Learning: Escaping the Impossibility of Group Fairness](https://mlsanity.com/paper/2405.20447/) (2024)
- [Decision-Making under Miscalibration](https://mlsanity.com/paper/2203.09852/) (2022)
- [Retiring DeltaDP: New Distribution-Level Metrics for Demographic Parity](https://mlsanity.com/paper/2301.13443/) (2023)
- [Priority-based Post-Processing Bias Mitigation for Individual and Group Fairness](https://mlsanity.com/paper/2102.00417/) (2021)
- [Practical Bias Mitigation through Proxy Sensitive Attribute Label Generation](https://mlsanity.com/paper/2312.15994/) (2023)
- [Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection](https://mlsanity.com/paper/2502.04528/) (2025)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [Fair Decisions from Calibrated Scores: Achieving Optimal Classification While Satisfying Sufficiency](https://mlsanity.com/paper/2602.07285/) (2026)

## Abstract (arXiv)

A popular methodology for building binary decision-making classifiers in the presence of imperfect information is to first construct a non-binary "scoring" classifier that is calibrated over all protected groups, and then to post-process this score to obtain a binary decision. We study the feasibility of achieving various fairness properties by post-processing calibrated scores, and then show that deferring post-processors allow for more fairness conditions to hold on the final decision. Specifically, we show: 1. There does not exist a general way to post-process a calibrated classifier to equalize protected groups' positive or negative predictive value (PPV or NPV). For certain "nice" calibrated classifiers, either PPV or NPV can be equalized when the post-processor uses different thresholds across protected groups, though there exist distributions of calibrated scores for which the two measures cannot be both equalized. When the post-processing consists of a single global threshold across all groups, natural fairness properties, such as equalizing PPV in a nontrivial way, do not hold even for "nice" classifiers. 2. When the post-processing is allowed to `defer' on some decisions (that is, to avoid making a decision by handing off some examples to a separate process), then for the non-deferred decisions, the resulting classifier can be made to equalize PPV, NPV, false positive rate (FPR) and false negative rate (FNR) across the protected groups. This suggests a way to partially evade the impossibility results of Chouldechova and Kleinberg et al., which preclude equalizing all of these measures simultaneously. We also present different deferring strategies and show how they affect the fairness properties of the overall system. We evaluate our post-processing techniques using the COMPAS data set from 2016.