# When is it Better to Compare than to Score?

arXiv: https://arxiv.org/abs/1406.6618 · HTML: https://mlsanity.com/paper/1406.6618/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2014.

## Built on by (most cited first, 9 of 9)

- [ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities](https://mlsanity.com/paper/2412.06745/) (2024)
- [Bayesian Optimization Augmented with Actively Elicited Expert Knowledge](https://mlsanity.com/paper/2208.08742/) (2022)
- [Efficient Exploration in Binary and Preferential Bayesian Optimization](https://mlsanity.com/paper/2110.09361/) (2021)
- [Audience-Centric Natural Language Generation via Style Infusion](https://mlsanity.com/paper/2301.10283/) (2023)
- [Evaluating Methods for Ground-Truth-Free Foreign Accent Conversion](https://mlsanity.com/paper/2309.02133/) (2023)
- [From User Preferences to Base Score Extraction Functions in Gradual Argumentation (with Appendix)](https://mlsanity.com/paper/2602.14674/) (2026)
- [Sorting from Counterexamples](https://mlsanity.com/paper/2608.21579/) (2026)
- [Automatic design optimization of preference-based subjective evaluation with online learning in crowdsourcing environment](https://mlsanity.com/paper/2403.06100/) (2024)
- [Elicitation-Augmented Bayesian Optimization](https://mlsanity.com/paper/2605.12079/) (2026)

## Abstract (arXiv)

When eliciting judgements from humans for an unknown quantity, one often has the choice of making direct-scoring (cardinal) or comparative (ordinal) measurements. In this paper we study the relative merits of either choice, providing empirical and theoretical guidelines for the selection of a measurement scheme. We provide empirical evidence based on experiments on Amazon Mechanical Turk that in a variety of tasks, (pairwise-comparative) ordinal measurements have lower per sample noise and are typically faster to elicit than cardinal ones. Ordinal measurements however typically provide less information. We then consider the popular Thurstone and Bradley-Terry-Luce (BTL) models for ordinal measurements and characterize the minimax error rates for estimating the unknown quantity. We compare these minimax error rates to those under cardinal measurement models and quantify for what noise levels ordinal measurements are better. Finally, we revisit the data collected from our experiments and show that fitting these models confirms this prediction: for tasks where the noise in ordinal measurements is sufficiently low, the ordinal approach results in smaller errors in the estimation.