# Decreasing Annotation Burden of Pairwise Comparisons with Human-in-the-Loop Sorting: Application in Medical Image Artifact Rating

arXiv: https://arxiv.org/abs/2202.04823 · HTML: https://mlsanity.com/paper/2202.04823/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 22 references that are in the map.
- First posted on arXiv 10 Feb 2022.

## Builds on (most cited first, 3 of 3)

- [Simple, Robust and Optimal Ranking from Pairwise Comparisons](https://mlsanity.com/paper/1512.08949/) (2015)
- [Hybrid-MST: A Hybrid Active Sampling Strategy for Pairwise Preference Aggregation](https://mlsanity.com/paper/1810.08851/) (2018)
- [Accelerated Experimental Design for Pairwise Comparisons](https://mlsanity.com/paper/1901.06080/) (2019)

## Built on by (most cited first, 5 of 5)

- [Active Preference Learning for Ordering Items In- and Out-of-sample](https://mlsanity.com/paper/2405.03059/) (2024)
- [EZ-Sort: Efficient Pairwise Comparison via Zero-Shot CLIP-Based Pre-Ordering and Human-in-the-Loop Sorting](https://mlsanity.com/paper/2508.21550/) (2025)
- [Dodgersort: Uncertainty-Aware VLM-Guided Human-in-the-Loop Pairwise Ranking](https://mlsanity.com/paper/2603.20839/) (2026)
- [MetaRanker: Human-in-the-loop Active Ranking for Metalens Image Quality](https://mlsanity.com/paper/2605.29212/) (2026)
- [Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling](https://mlsanity.com/paper/2606.15623/) (2026)

## Abstract (arXiv)

Ranking by pairwise comparisons has shown improved reliability over ordinal classification. However, as the annotations of pairwise comparisons scale quadratically, this becomes less practical when the dataset is large. We propose a method for reducing the number of pairwise comparisons required to rank by a quantitative metric, demonstrating the effectiveness of the approach in ranking medical images by image quality in this proof of concept study. Using the medical image annotation software that we developed, we actively subsample pairwise comparisons using a sorting algorithm with a human rater in the loop. We find that this method substantially reduces the number of comparisons required for a full ordinal ranking without compromising inter-rater reliability when compared to pairwise comparisons without sorting.