mlsanityData to 7 Oct 2026

Conformal Prediction Sets Can Cause Disparate Impact

arXiv 2410.01888 ↗First posted 2 Oct 2024cs.LG stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

Conformal prediction is a statistically rigorous method for quantifying uncertainty in models by having them output sets of predictions, with larger sets indicating more uncertainty. However, prediction sets are not inherently actionable; many applications require a single output to act on, not several. To overcome this limitation, prediction sets can be provided to a human who then makes an informed decision. In any such system it is crucial to ensure the fairness of outcomes across protected groups, and researchers have proposed that Equalized Coverage be used as the standard for fairness. By conducting experiments with human participants, we demonstrate that providing prediction sets can lead to disparate impact in decisions. Disquietingly, we find that providing sets that satisfy Equalized Coverage actually increases disparate impact compared to marginal coverage. Instead of equalizing coverage, we propose to equalize set sizes across groups which empirically leads to lower disparate impact.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 40 of its 69 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

40 of 69 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  8. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  9. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  10. 2017Counterfactual Fairnesscited by 802
  11. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  12. 2014Certifying and removing disparate impactcited by 715
28 more
  1. 2019Conformalized Quantile Regressioncited by 620
  2. 2017On Fairness and Calibrationcited by 406
  3. 2020Classification with Valid and Adaptive Coveragecited by 390
  4. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  5. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  6. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 199
  7. 2023Conformal Prediction With Conditional Guaranteescited by 146
  8. 2024Language Models with Conformal Factuality Guaranteescited by 128
  9. 2023Class-Conditional Conformal Prediction with Many Classescited by 109
  10. 2021Fair Conformal Predictors for Applications in Medical Imagingcited by 74
  11. 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
  12. 2023Conformal Prediction for Deep Classifier via Label Rankingcited by 52
  13. 2021Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Datacited by 48
  14. 2019With Malice Towards None: Assessing Uncertainty via Equalized Coveragecited by 43
  15. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  16. 2022Disparate Impact in Differential Privacy from Gradient Misalignmentcited by 31
  17. 2021Set-valued classification -- overview via a unified frameworkcited by 27
  18. 2023Designing Decision Support Systems Using Counterfactual Prediction Setscited by 26
  19. 2024Towards Human-AI Complementarity with Prediction Setscited by 25
  20. 2022Conformalized Fairness via Quantile Regressioncited by 19
  21. 2021Loss-Aversively Fair Classificationcited by 18
  22. 2023HappyMap: A Generalized Multi-calibration Methodcited by 17
  23. 2023Equal Opportunity of Coverage in Fair Regressioncited by 16
  24. 2023Evaluating the Fairness of Deep Learning Uncertainty Estimates in Medical Image Analysiscited by 15
  25. 2021Nested Conformal Prediction Sets for Classification with Applications to Probation Datacited by 12
  26. 2024Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labelingcited by 12
  27. 2024Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Setscited by 9
  28. 2021Classification Under Ambiguity: When Is Average-K Better Than Top-K?cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  2. 2025Conformal Prediction and Human Decision Makingcited by 8
  3. 2025Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factualitycited by 8
  4. 2025Document Summarization with Conformal Importance Guaranteescited by 6
  5. 2025Trustworthy AI Must Account for Interactionscited by 4
  6. 2025LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systemscited by 4
  7. 2025FedCF: Fair Federated Conformal Predictioncited by 3
  8. 2026Conformal Agent Error Attributioncited by 3
  9. 2025Counterfactually Fair Conformal Predictioncited by 2
  10. 2026Beyond Procedure: Substantive Fairness in Conformal Predictioncited by 2
  11. 2026On the Burden of Achieving Fairness in Conformal Predictioncited by 1
  12. 2026Robust LLM Performance Certification via Constrained Maximum Likelihood Estimationcited by 0
2 more
  1. 2026Unifying Conformal Language Tasks with In-Context Ensemblescited by 0
  2. 2026SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictionscited by 0