mlsanityData to 7 Oct 2026

On the Utility of Prediction Sets in Human-AI Teams

arXiv 2205.01411 ↗First posted 3 May 2022cs.AI cs.HCcited by 34

Not on the map. Why?

Abstract (arXiv)

Research on human-AI teams usually provides experts with a single label, which ignores the uncertainty in a model's recommendation. Conformal prediction (CP) is a well established line of research that focuses on building a theoretically grounded, calibrated prediction set, which may contain multiple labels. We explore how such prediction sets impact expert decision-making in human-AI teams. Our evaluation on human subjects finds that set valued predictions positively impact experts. However, we notice that the predictive sets provided by CP can be very large, which leads to unhelpful AI assistants. To mitigate this, we introduce D-CP, a method to perform CP on some examples and defer to experts. We prove that D-CP can reduce the prediction set size of non-deferred examples. We show how D-CP performs in quantitative and in human subject experiments (n=120). Our results suggest that CP prediction sets improve human-AI team performance over showing the top-1 prediction alone, and that experts find D-CP prediction sets are more useful than CP prediction sets.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

15 of 30 references.

References · 15

  1. 2016Wide Residual Networkscited by 3,057
  2. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  3. 2016Distribution-Free Predictive Inference For Regressioncited by 657
  4. 2019Conformal Prediction Under Covariate Shiftcited by 511
  5. 2020Classification with Valid and Adaptive Coveragecited by 390
  6. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  7. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 270
  8. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  9. 2019Human uncertainty makes classification more robustcited by 231
  10. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  11. 2020Learning to Complement Humanscited by 123
  12. 2021Learning Optimal Conformal Classifierscited by 108
3 more
  1. 2021Differentiable Learning Under Triagecited by 74
  2. 2021Role of Human-AI Interaction in Selective Predictioncited by 33
  3. 2021Optimized conformal classification using gradient descent approximationcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 10; 2025: 7; 2026: 7.

34 papers.

Sort:

By area:

  1. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  2. 2022Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensemblescited by 59
  3. 2023Federated Conformal Predictors for Distributed Uncertainty Quantificationcited by 46
  4. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  5. 2024Length Optimization in Conformal Predictioncited by 39
  6. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  7. 2023Designing Decision Support Systems Using Counterfactual Prediction Setscited by 26
  8. 2024Towards Human-AI Complementarity with Prediction Setscited by 25
  9. 2024Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibrationcited by 18
  10. 2023Learning Personalized Decision Support Policiescited by 13
  11. 2024Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labelingcited by 12
  12. 2022User-Interactive Offline Reinforcement Learningcited by 11
22 more
  1. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
  2. 2024Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Setscited by 9
  3. 2025Direct Prediction Set Minimization via Bilevel Conformal Classifier Trainingcited by 8
  4. 2024Learning to Complement and to Defer to Multiple Userscited by 7
  5. 2025Human-AI Collaborative Uncertainty Quantificationcited by 6
  6. 2023Learning to Complement with Multiple Humanscited by 5
  7. 2023When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-makingcited by 4
  8. 2025Test-time augmentation improves efficiency in conformal predictioncited by 3
  9. 2026Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformerscited by 3
  10. 2023RR-CP: Reliable-Region-Based Conformal Prediction for Trustworthy Medical Image Classificationcited by 2
  11. 2025No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Predictioncited by 2
  12. 2026Multi-Round Human-AI Collaboration with User-Specified Requirementscited by 2
  13. 2024A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessmentcited by 1
  14. 2025Uncertainty Guarantees on Automated Precision Weeding using Conformal Predictioncited by 1
  15. 2026Strategic Decision Support for AI Agentscited by 1
  16. 2026What Types of Human-AI Teams Exist?cited by 1
  17. 2024Fundus Image-based Visual Acuity Assessment with PAC-Guaranteescited by 0
  18. 2025Cost-Sensitive Conformal Training with Provably Controllable Learning Boundscited by 0
  19. 2025Conformal Correction for Efficiency May be at Odds with Entropycited by 0
  20. 2026PLACO: A Multi-Stage Framework for Cost-Effective Performance in Human-AI Teamscited by 0
  21. 2026Adaptive Conformal Prediction for Reliable and Explainable Medical Image Classificationcited by 0
  22. 2026Set-Valued Policy Learningcited by 0