# Using tournaments to calculate AUROC for zero-shot classification with LLMs

arXiv: https://arxiv.org/abs/2502.15018 · HTML: https://mlsanity.com/paper/2502.15018/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 13 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 3 of 3)

- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Position Paper On Diagnostic Uncertainty Estimation from Large Language Models: Next-Word Probability Is Not Pre-test Probability](https://mlsanity.com/paper/2411.04962/) (2024)

## Built on by (most cited first, 2 of 2)

- [AGENT-X: Adaptive Guideline-based Expert Network for Threshold-free AI-generated teXt detection](https://mlsanity.com/paper/2505.15261/) (2025)
- [CLSGen: A Dual-Head Fine-Tuning Framework for Joint Probabilistic Classification and Verbalized Explanation](https://mlsanity.com/paper/2604.11801/) (2026)

## Abstract (arXiv)

Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset, using LLMs to produce relative rankings of those instances. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.