mlsanityData to 7 Oct 2026

Fairness Evaluation in Presence of Biased Noisy Labels

arXiv 2003.13808 ↗First posted 30 Mar 2020stat.ME cs.CYcited by 38

Not on the map. Why?

Abstract (arXiv)

Risk assessment tools are widely used around the country to inform decision making within the criminal justice system. Recently, considerable attention has been devoted to the question of whether such tools may suffer from racial bias. In this type of assessment, a fundamental issue is that the training and evaluation of the model is based on a variable (arrest) that may represent a noisy version of an unobserved outcome of more central interest (offense). We propose a sensitivity analysis framework for assessing how assumptions on the noise across groups affect the predictive bias properties of the risk assessment model as a predictor of reoffense. Our experimental results on two real world criminal justice data sets demonstrate how even small biases in the observed labels may call into question the conclusions of an analysis based on the noisy outcome.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 8; 2023: 12; 2024: 5; 2025: 3; 2026: 3.

38 papers.

Sort:

  1. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  2. 2021Fairness without Imputation: A Decision Tree Approach for Fair Prediction with Missing Valuescited by 24
  3. 2021Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systemscited by 20
  4. 2023Aleatoric and Epistemic Discrimination: Fundamental Limits of Fairness Interventionscited by 19
  5. 2021Fairness-Aware PAC Learning from Corrupted Datacited by 18
  6. 2022Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributescited by 17
  7. 2022What's the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatmentcited by 14
  8. 2021Can Less be More? When Increasing-to-Balancing Label Noise Rates Considered Beneficialcited by 13
  9. 2021The Impact of Algorithmic Risk Assessments on Human Predictions and its Analysis via Crowdsourcing Studiescited by 13
  10. 2023On (assessing) the fairness of risk score modelscited by 12
  11. 2023Counterfactual Prediction Under Outcome Measurement Errorcited by 10
  12. 2021Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approachcited by 9
26 more
  1. 2024A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learningcited by 9
  2. 2024A structured regression approach for evaluating model performance across intersectional subgroupscited by 8
  3. 2024The Fragility of Fairness: Causal Sensitivity Analysis for Fair Machine Learningcited by 8
  4. 2021Measuring Model Fairness under Noisy Covariates: A Theoretical Perspectivecited by 7
  5. 2022Understanding Unfairness in Fraud Detection through Model and Data Bias Interactionscited by 7
  6. 2023Quantifying and mitigating the impact of label errors on model disparity metricscited by 7
  7. 2024Improving LLM Group Fairness on Tabular Data via In-Context Learningcited by 6
  8. 2022Social Bias Meets Data Bias: The Impacts of Labeling and Measurement Errors on Fairness Criteriacited by 5
  9. 2023How Far Can Fairness Constraints Help Recover From Biased Data?cited by 5
  10. 2022Prisoners of Their Own Devices: How Models Induce Data Bias in Performative Predictioncited by 4
  11. 2023Weak Supervision Performance Evaluation via Partial Identificationcited by 4
  12. 2022More Data Can Lead Us Astray: Active Data Acquisition in the Presence of Label Biascited by 3
  13. 2025Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrestcited by 3
  14. 2022A Survey on Preserving Fairness Guarantees in Changing Environmentscited by 2
  15. 2023Systematic analysis of the impact of label noise correction on ML Fairnesscited by 2
  16. 2023Mitigating Label Bias in Machine Learning: Fairness through Confident Learningcited by 2
  17. 2024Predictive Performance Comparison of Decision Policies Under Confoundingcited by 2
  18. 2022I Prefer not to Say: Protecting User Consent in Models with Optional Personal Datacited by 1
  19. 2023On Comparing Fair Classifiers under Data Biascited by 1
  20. 2023Mitigating Label Bias via Decoupled Confident Learningcited by 1
  21. 2023On the Cause of Unfairness: A Training Sample Perspectivecited by 0
  22. 2025GFLC: Graph-based Fairness-aware Label Correction for Fair Classificationcited by 0
  23. 2025Bias-Aware Mislabeling Detection via Decoupled Confident Learningcited by 0
  24. 2026No evaluation without fair representation : Impact of label and selection bias on the evaluation, performance and mitigation of classification modelscited by 0
  25. 2026To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systemscited by 0
  26. 2026Nous: Learning and Certifying Memory Decisions Before Source Calibrationcited by 0