# Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons

arXiv: https://arxiv.org/abs/2608.10045 · HTML: https://mlsanity.com/paper/2608.10045/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 11 of 11)

- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-tuning language models to find agreement among humans with diverse preferences](https://mlsanity.com/paper/2211.15006/) (2022)
- [Non-convex Optimization for Machine Learning](https://mlsanity.com/paper/1712.07897/) (2017)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach](https://mlsanity.com/paper/1609.03240/) (2016)
- [The Expertise Problem: Learning from Specialized Feedback](https://mlsanity.com/paper/2211.06519/) (2022)
- [Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction](https://mlsanity.com/paper/1606.05374/) (2016)
- [Rank Aggregation via Heterogeneous Thurstone Preference Models](https://mlsanity.com/paper/1912.01211/) (2019)
- [IMDB-WIKI-SbS: An Evaluation Dataset for Crowdsourced Pairwise Comparisons](https://mlsanity.com/paper/2110.14990/) (2021)
- [False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking](https://mlsanity.com/paper/1605.05860/) (2016)
- [Aggregation of pairwise comparisons with reduction of biases](https://mlsanity.com/paper/1906.03711/) (2019)

## Abstract (arXiv)

The problem of learning from pairwise comparisons has been widely studied across many domains such as recommendation systems, social choice, and more recently, fine-tuning large language models. In this problem, the goal is to learn item rewards based on pairwise comparisons between them. In many scenarios, these comparisons are elicited from crowdworkers using platforms such as Amazon Mechanical Turk, Scale AI, etc. However, crowdworkers are often unreliable due to limited domain knowledge or revenue-maximizing (spamming) behavior. In this work, our goal is to understand whether worker reliability (competency) can be learned jointly with item rewards. To this end, we adopt the Boltzmann-rational model for pairwise comparisons, which extends the Bradley-Terry-Luce model by incorporating worker competencies. We derive an EM-based algorithm for learning under this model by introducing Polya-Gamma latent variables to transform the logistic likelihood into a conditionally Gaussian form, enabling tractable optimization and leading to a simplified Q function in the E-step of the algorithm. This technique allows us to reduce our formulation to a matrix sensing problem, using which we establish theoretical convergence guarantees for our algorithm. We conduct extensive experiments on real-world and synthetic datasets. These experiments demonstrate the advantages of using our algorithm over several baselines and confirm its strong robustness to both spammers and adversarial workers, highlighting its practical effectiveness in realistic crowdsourcing and reward learning settings. The code and data is publicly available at https://github.com/KaustubhShejole/BoRaEM.