# Two-stage Risk Control with Application to Ranked Retrieval

arXiv: https://arxiv.org/abs/2404.17769 · HTML: https://mlsanity.com/paper/2404.17769/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Builds on (most cited first, 9 of 9)

- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [A Conformal Prediction Approach to Explore Functional Data](https://mlsanity.com/paper/1302.6452/) (2013)
- [Recommendation Systems with Distribution-Free Reliability Guarantees](https://mlsanity.com/paper/2207.01609/) (2022)
- [Uncertainty Quantification for Fairness in Two-Stage Recommender Systems](https://mlsanity.com/paper/2205.15436/) (2022)
- [Inference-time Stochastic Ranking with Risk Control](https://mlsanity.com/paper/2306.07188/) (2023)

## Built on by (most cited first, 6 of 6)

- [Selective Conformal Risk Control](https://mlsanity.com/paper/2512.12844/) (2025)
- [A Joint Finite-Sample Certificate for Adaptive Selective Conformal Risk Control](https://mlsanity.com/paper/2606.08517/) (2026)
- [Uncertainty Quantification and Data Efficiency in AI: An Information-Theoretic Perspective](https://mlsanity.com/paper/2512.05267/) (2025)
- [Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better](https://mlsanity.com/paper/2601.21455/) (2026)
- [CVE-SAI: Counterfactual Visual Evidence-Guided Selective Attribute Indexing for Risk-Controlled E-commerce Search](https://mlsanity.com/paper/2608.25023/) (2026)
- [Language Distances are Practical for Equitable Cross-Lingual Transfer](https://mlsanity.com/paper/2609.32331/) (2026)

## Abstract (arXiv)

Practical machine learning systems often operate in multiple sequential stages, as seen in ranking and recommendation systems, which typically include a retrieval phase followed by a ranking phase. Effectively assessing prediction uncertainty and ensuring effective risk control in such systems pose significant challenges due to their inherent complexity. To address these challenges, we developed two-stage risk control methods based on the recently proposed learn-then-test (LTT) and conformal risk control (CRC) frameworks. Unlike the methods in prior work that address multiple risks, our approach leverages the sequential nature of the problem, resulting in reduced computational burden. We provide theoretical guarantees for our proposed methods and design novel loss functions tailored for ranked retrieval tasks. The effectiveness of our approach is validated through experiments on two large-scale, widely-used datasets: MSLR-Web and Yahoo LTRC.