# Improving Screening Processes via Calibrated Subset Selection arXiv: https://arxiv.org/abs/2202.01147 ยท HTML: https://mlsanity.com/paper/2202.01147/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 56 references that are in the map. - First posted on arXiv 2 Feb 2022. ## Builds on (most cited first, 25 of 25) - [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017) - [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016) - [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021) - [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016) - [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019) - [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016) - [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021) - [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018) - [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017) - [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017) - [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021) - [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019) - [Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search](https://mlsanity.com/paper/1905.01989/) (2019) - [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021) - [Distribution-free binary classification: prediction sets, confidence intervals and calibration](https://mlsanity.com/paper/2006.10564/) (2020) - [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021) - [Moment Multicalibration for Uncertainty Estimation](https://mlsanity.com/paper/2008.08037/) (2020) - [Individual Calibration with Randomized Forecasting](https://mlsanity.com/paper/2006.10288/) (2020) - [Distribution-free calibration guarantees for histogram binning without sample splitting](https://mlsanity.com/paper/2105.04656/) (2021) - [Decisions, Counterfactual Explanations and Strategic Behavior](https://mlsanity.com/paper/2002.04333/) (2020) - [What does it mean to solve the problem of discrimination in hiring? Social, technical and legal perspectives from the UK on automated hiring systems](https://mlsanity.com/paper/1910.06144/) (2019) - [Set-valued classification -- overview via a unified framework](https://mlsanity.com/paper/2102.12318/) (2021) - [Monotonic classification: an overview on algorithms, performance measures and data sets](https://mlsanity.com/paper/1811.07155/) (2018) - [Fair Allocation through Selective Information Acquisition](https://mlsanity.com/paper/1911.02715/) (2019) - [On component interactions in two-stage recommender systems](https://mlsanity.com/paper/2106.14979/) (2021) ## Built on by (most cited first, 16 of 16) - [Selection by Prediction with Conformal p-values](https://mlsanity.com/paper/2210.01408/) (2022) - [Human-Aligned Calibration for AI-Assisted Decision Making](https://mlsanity.com/paper/2306.00074/) (2023) - [Uncertainty Quantification for Fairness in Two-Stage Recommender Systems](https://mlsanity.com/paper/2205.15436/) (2022) - [Classification Under Strategic Self-Selection](https://mlsanity.com/paper/2402.15274/) (2024) - [Impact of Decentralized Learning on Player Utilities in Stackelberg Games](https://mlsanity.com/paper/2403.00188/) (2024) - [Human-Alignment Influences the Utility of AI-assisted Decision Making](https://mlsanity.com/paper/2501.14035/) (2025) - [When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-making](https://mlsanity.com/paper/2308.11721/) (2023) - [Fairness in Ranking under Disparate Uncertainty](https://mlsanity.com/paper/2309.01610/) (2023) - [Context matters for fairness -- a case study on the effect of spatial distribution shifts](https://mlsanity.com/paper/2206.11436/) (2022) - [Inference-time Stochastic Ranking with Risk Control](https://mlsanity.com/paper/2306.07188/) (2023) - [Multi-Round Human-AI Collaboration with User-Specified Requirements](https://mlsanity.com/paper/2602.17646/) (2026) - [On the Within-Group Fairness of Screening Classifiers](https://mlsanity.com/paper/2302.00025/) (2023) - [Reliable algorithm selection for machine learning-guided design](https://mlsanity.com/paper/2503.20767/) (2025) - [Learning to Decide with AI Assistance under Human-Alignment](https://mlsanity.com/paper/2605.12646/) (2026) - [Strategic Decision Support for AI Agents](https://mlsanity.com/paper/2606.12587/) (2026) - [Ranking with Slot Constraints](https://mlsanity.com/paper/2310.17870/) (2023) ## Abstract (arXiv) Many selection processes such as finding patients qualifying for a medical trial or retrieval pipelines in search engines consist of multiple stages, where an initial screening stage focuses the resources on shortlisting the most promising candidates. In this paper, we investigate what guarantees a screening classifier can provide, independently of whether it is constructed manually or trained. We find that current solutions do not enjoy distribution-free theoretical guarantees -- we show that, in general, even for a perfectly calibrated classifier, there always exist specific pools of candidates for which its shortlist is suboptimal. Then, we develop a distribution-free screening algorithm -- called Calibrated Subset Selection (CSS) -- that, given any classifier and some amount of calibration data, finds near-optimal shortlists of candidates that contain a desired number of qualified candidates in expectation. Moreover, we show that a variant of CSS that calibrates a given classifier multiple times across specific groups can create shortlists with provable diversity guarantees. Experiments on US Census survey data validate our theoretical results and show that the shortlists provided by our algorithm are superior to those provided by several competitive baselines.