# On Minimizing Cost in Legal Document Review Workflows

arXiv: https://arxiv.org/abs/2106.09866 · HTML: https://mlsanity.com/paper/2106.09866/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2021.

## Built on by (most cited first, 5 of 5)

- [HC4: A New Suite of Test Collections for Ad Hoc CLIR](https://mlsanity.com/paper/2201.09992/) (2022)
- [Certifying One-Phase Technology-Assisted Reviews](https://mlsanity.com/paper/2108.12746/) (2021)
- [Impact of Stop Sets on Stopping Active Learning for Text Classification](https://mlsanity.com/paper/2201.05460/) (2022)
- [Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection](https://mlsanity.com/paper/2309.06131/) (2023)
- [Opening the TAR Black Box: Developing an Interpretable System for eDiscovery Using the Fuzzy ARTMAP Neural Network](https://mlsanity.com/paper/2305.04237/) (2023)

## Abstract (arXiv)

Technology-assisted review (TAR) refers to human-in-the-loop machine learning workflows for document review in legal discovery and other high recall review tasks. Attorneys and legal technologists have debated whether review should be a single iterative process (one-phase TAR workflows) or whether model training and review should be separate (two-phase TAR workflows), with implications for the choice of active learning algorithm. The relative cost of manual labeling for different purposes (training vs. review) and of different documents (positive vs. negative examples) is a key and neglected factor in this debate. Using a novel cost dynamics analysis, we show analytically and empirically that these relative costs strongly impact whether a one-phase or two-phase workflow minimizes cost. We also show how category prevalence, classification task difficulty, and collection size impact the optimal choice not only of workflow type, but of active learning method and stopping point.