# Heuristic Stopping Rules For Technology-Assisted Review

arXiv: https://arxiv.org/abs/2106.09871 · HTML: https://mlsanity.com/paper/2106.09871/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 38 references that are in the map.
- First posted on arXiv 18 Jun 2021.

## Builds on (most cited first, 3 of 3)

- [Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review](https://mlsanity.com/paper/1504.06868/) (2015)
- [libact: Pool-based Active Learning in Python](https://mlsanity.com/paper/1710.00379/) (2017)
- [Providing More Efficient Access To Government Records: A Use Case Involving Application of Machine Learning to Improve FOIA Review for the Deliberative Process Privilege](https://mlsanity.com/paper/2011.07203/) (2020)

## Built on by (most cited first, 2 of 2)

- [Certifying One-Phase Technology-Assisted Reviews](https://mlsanity.com/paper/2108.12746/) (2021)
- [Finite-Sample Coverage Audits for High-Recall Candidate Generation: Certification and Learning-Theoretic Design](https://mlsanity.com/paper/2607.21480/) (2026)

## Abstract (arXiv)

Technology-assisted review (TAR) refers to human-in-the-loop active learning workflows for finding relevant documents in large collections. These workflows often must meet a target for the proportion of relevant documents found (i.e. recall) while also holding down costs. A variety of heuristic stopping rules have been suggested for striking this tradeoff in particular settings, but none have been tested against a range of recall targets and tasks. We propose two new heuristic stopping rules, Quant and QuantCI based on model-based estimation techniques from survey research. We compare them against a range of proposed heuristics and find they are accurate at hitting a range of recall targets while substantially reducing review costs.