# Cost-Effective HITs for Relative Similarity Comparisons

arXiv: https://arxiv.org/abs/1404.3291 · HTML: https://mlsanity.com/paper/1404.3291/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2014.

## Built on by (most cited first, 14 of 14)

- [Searching for Structure in Unfalsifiable Claims](https://mlsanity.com/paper/2209.00495/) (2022)
- [Active metric learning and classification using similarity queries](https://mlsanity.com/paper/2202.01953/) (2022)
- [Visual Knowledge Tracing](https://mlsanity.com/paper/2207.10157/) (2022)
- [Active Learning of Ordinal Embeddings: A User Study on Football Data](https://mlsanity.com/paper/2207.12710/) (2022)
- [Trustworthy Human Computation: A Survey](https://mlsanity.com/paper/2210.12324/) (2022)
- [A Unified Batch Selection Policy for Active Metric Learning](https://mlsanity.com/paper/2102.07365/) (2021)
- [Elliptical Ordinal Embedding](https://mlsanity.com/paper/2105.10457/) (2021)
- [Comparing Apples to Oranges: Learning Similarity Functions for Data Produced by Different Distributions](https://mlsanity.com/paper/2208.12731/) (2022)
- [A Revenue Function for Comparison-Based Hierarchical Clustering](https://mlsanity.com/paper/2211.16459/) (2022)
- [Fast Interactive Search with a Scale-Free Comparison Oracle](https://mlsanity.com/paper/2306.01814/) (2023)
- [Metric Learning in an RKHS](https://mlsanity.com/paper/2508.04476/) (2025)
- [Measuring IIA Violations in Similarity Choices with Bayesian Models](https://mlsanity.com/paper/2508.14615/) (2025)
- [LORE: Jointly Learning the Intrinsic Dimensionality and Relative Similarity Structure From Ordinal Data](https://mlsanity.com/paper/2602.04192/) (2026)
- [Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization](https://mlsanity.com/paper/2610.14757/) (2026)

## Abstract (arXiv)

Similarity comparisons of the form "Is object a more similar to b than to c?" are useful for computer vision and machine learning applications. Unfortunately, an embedding of n points is specified by n3 triplets, making collecting every triplet an expensive task. In noticing this difficulty, other researchers have investigated more intelligent triplet sampling techniques, but they do not study their effectiveness or their potential drawbacks. Although it is important to reduce the number of collected triplets, it is also important to understand how best to display a triplet collection task to a user. In this work we explore an alternative display for collecting triplets and analyze the monetary cost and speed of the display. We propose best practices for creating cost effective human intelligence tasks for collecting triplets. We show that rather than changing the sampling algorithm, simple changes to the crowdsourcing UI can lead to much higher quality embeddings. We also provide a dataset as well as the labels collected from crowd workers.