# Certifying One-Phase Technology-Assisted Reviews

arXiv: https://arxiv.org/abs/2108.12746 · HTML: https://mlsanity.com/paper/2108.12746/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 59 references that are in the map.
- First posted on arXiv 29 Aug 2021.

## Builds on (most cited first, 3 of 3)

- [Limitations of Assessing Active Learning Performance at Runtime](https://mlsanity.com/paper/1901.10338/) (2019)
- [On Minimizing Cost in Legal Document Review Workflows](https://mlsanity.com/paper/2106.09866/) (2021)
- [Heuristic Stopping Rules For Technology-Assisted Review](https://mlsanity.com/paper/2106.09871/) (2021)

## Built on by (most cited first, 2 of 2)

- [How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond](https://mlsanity.com/paper/2501.05714/) (2025)
- [Finite-Sample Coverage Audits for High-Recall Candidate Generation: Certification and Learning-Theoretic Design](https://mlsanity.com/paper/2607.21480/) (2026)

## Abstract (arXiv)

Technology-assisted review (TAR) workflows based on iterative active learning are widely used in document review applications. Most stopping rules for one-phase TAR workflows lack valid statistical guarantees, which has discouraged their use in some legal contexts. Drawing on the theory of quantile estimation, we provide the first broadly applicable and statistically valid sample-based stopping rules for one-phase TAR. We further show theoretically and empirically that overshooting a recall target, which has been treated as innocuous or desirable in past evaluations of stopping rules, is a major source of excess cost in one-phase TAR workflows. Counterintuitively, incurring a larger sampling cost to reduce excess recall leads to lower total cost in almost all scenarios.