# Optimal Fair Aggregation of Crowdsourced Noisy Labels using Demographic Parity Constraints

arXiv: https://arxiv.org/abs/2601.23221 · HTML: https://mlsanity.com/paper/2601.23221/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [Fair and Optimal Classification via Post-Processing](https://mlsanity.com/paper/2211.01528/) (2022)
- [Bayes-Optimal Classifiers under Group Fairness](https://mlsanity.com/paper/2202.09724/) (2022)
- [Consistency of weighted majority votes](https://mlsanity.com/paper/1312.0451/) (2013)
- [Exact Exponent in Optimal Rates for Crowdsourcing](https://mlsanity.com/paper/1605.07696/) (2016)
- [Crowdsourcing with Fairness, Diversity and Budget Constraints](https://mlsanity.com/paper/1810.13314/) (2018)
- [The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learning](https://mlsanity.com/paper/2410.09600/) (2024)
- [Learning from Noisy Labels via Conditional Distributionally Robust Optimization](https://mlsanity.com/paper/2411.17113/) (2024)
- [Fairness and Bias in Truth Discovery Algorithms: An Experimental Analysis](https://mlsanity.com/paper/2304.12573/) (2023)
- [Learning From Crowdsourced Noisy Labels: A Signal Processing Perspective](https://mlsanity.com/paper/2407.06902/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection](https://mlsanity.com/paper/2607.24622/) (2026)
- [On the Calibration of Crowdsourcing Aggregators](https://mlsanity.com/paper/2610.16815/) (2026)

## Abstract (arXiv)

As acquiring reliable ground-truth labels is usually costly, or infeasible, crowdsourcing and aggregation of noisy human annotations is the typical resort. Aggregating subjective labels, though, may amplify individual biases, particularly regarding sensitive features, raising fairness concerns. Nonetheless, fairness in crowdsourced aggregation remains largely unexplored, with no existing convergence guarantees and only limited post-processing approaches for enforcing varepsilon-fairness under demographic parity. We address this gap by analyzing the fairness s of crowdsourced aggregation methods within the varepsilon-fairness framework, for Majority Vote and Optimal Bayesian aggregation. In the small-crowd regime, we derive an upper bound on the fairness gap of Majority Vote in terms of the fairness gaps of the individual annotators. We further show that the fairness gap of the aggregated consensus converges exponentially fast to that of the ground-truth under interpretable conditions. Since ground-truth itself may still be unfair, we generalize a state-of-the-art multiclass fairness post-processing algorithm from the continuous to the discrete setting, which enforces strict demographic parity constraints to any aggregation rule. Experiments on synthetic and real datasets demonstrate the effectiveness of our approach and corroborate the theoretical insights.