# Beyond Labels: Information-Efficient Human-in-the-Loop Learning using Ranking and Selection Queries

arXiv: https://arxiv.org/abs/2602.15738 · HTML: https://mlsanity.com/paper/2602.15738/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 47 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora](https://mlsanity.com/paper/1606.02820/) (2016)
- [Learning Safety Constraints for Large Language Models](https://mlsanity.com/paper/2505.24445/) (2025)
- [Active embedding search via noisy paired comparisons](https://mlsanity.com/paper/1905.04363/) (2019)
- [Active Ordinal Querying for Tuplewise Similarity Learning](https://mlsanity.com/paper/1910.04115/) (2019)
- [Much Ado About Time: Exhaustive Annotation of Temporal Data](https://mlsanity.com/paper/1607.07429/) (2016)
- [A Survey on Image Aesthetic Assessment](https://mlsanity.com/paper/2103.11616/) (2021)
- [Active Learning for Binary Classification with Abstention](https://mlsanity.com/paper/1906.00303/) (2019)
- [PrefGen: Preference Guided Image Generation with Relative Attributes](https://mlsanity.com/paper/2304.00185/) (2023)
- [Feedback Coding for Active Learning](https://mlsanity.com/paper/2103.00654/) (2021)

## Built on by (most cited first, 1 of 1)

- [From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP](https://mlsanity.com/paper/2605.25226/) (2026)

## Abstract (arXiv)

Integrating human expertise into machine learning systems often reduces the role of experts to labeling oracles, a paradigm that limits the amount of information exchanged and fails to capture the nuances of human judgment. We address this challenge by developing a human-in-the-loop framework to learn binary classifiers with rich query types, consisting of item ranking and exemplar selection. We first introduce probabilistic human response models for these rich queries motivated by the relationship experimentally observed between the perceived implicit score of an item and its distance to the unknown classifier. Using these models, we then design active learning algorithms that leverage the rich queries to increase the information gained per interaction. We provide theoretical bounds on sample complexity and develop a tractable and computationally efficient variational approximation. Through experiments with simulated annotators derived from crowdsourced word-sentiment and image-aesthetic datasets, we demonstrate significant reductions on sample complexity. We further extend active learning strategies to select queries that maximize information rate, explicitly balancing informational value against annotation cost. This algorithm in the word sentiment classification task reduces learning time by more than 57% compared to traditional label-only active learning.