# SP-Rank: A Dataset for Ranked Preferences with Secondary Information

arXiv: https://arxiv.org/abs/2601.05253 · HTML: https://mlsanity.com/paper/2601.05253/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 31 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Builds on (most cited first, 8 of 8)

- [Eliciting Expertise without Verification](https://mlsanity.com/paper/1802.08312/) (2018)
- [The Wisdom of the Crowd and Higher-Order Beliefs](https://mlsanity.com/paper/2102.02666/) (2021)
- [Calibrating "Cheap Signals" in Peer Review without a Prior](https://mlsanity.com/paper/2312.07269/) (2023)
- [Wisdom of the Crowd Voting: Truthful Aggregation of Voter Information and Preferences](https://mlsanity.com/paper/2108.03749/) (2021)
- [Surprisingly Popular Voting Recovers Rankings, Surprisingly!](https://mlsanity.com/paper/2105.09386/) (2021)
- [The Surprising Effectiveness of SP Voting with Partial Preferences](https://mlsanity.com/paper/2406.00870/) (2024)
- [Rank Aggregation Using Scoring Rules](https://mlsanity.com/paper/2209.08856/) (2022)
- [Machine Truth Serum](https://mlsanity.com/paper/1909.13004/) (2019)

## Abstract (arXiv)

We introduce SP-Rank, the first large-scale, publicly available dataset for benchmarking algorithms that leverage both first-order preferences and second-order predictions in ranking tasks. Each datapoint includes a personal vote (first-order signal) and a meta-prediction of how others will vote (second-order signal), allowing richer modeling than traditional datasets that capture only individual preferences. SP-Rank contains over 12,000 human-generated datapoints across three domains -- geography, movies, and paintings, and spans nine elicitation formats with varying subset sizes. This structure enables empirical analysis of preference aggregation when expert identities are unknown but presumed to exist, and individual votes represent noisy estimates of a shared ground-truth ranking. We benchmark SP-Rank by comparing traditional aggregation methods that use only first-order votes against SP-Voting, a second-order method that jointly reasons over both signals to infer ground-truth rankings. While SP-Rank also supports models that rely solely on second-order predictions, our benchmarks emphasize the gains from combining both signals. We evaluate performance across three core tasks: (1) full ground-truth rank recovery, (2) subset-level rank recovery, and (3) probabilistic modeling of voter behavior. Results show that incorporating second-order signals substantially improves accuracy over vote-only methods. Beyond social choice, SP-Rank supports downstream applications in learning-to-rank, extracting expert knowledge from noisy crowds, and training reward models in preference-based fine-tuning pipelines. We release the dataset, code, and baseline evaluations (available at https://github.com/amrit19/SP-Rank-Dataset ) to foster research in human preference modeling, aggregation theory, and human-AI alignment.