# Eliciting Expertise without Verification

arXiv: https://arxiv.org/abs/1802.08312 · HTML: https://mlsanity.com/paper/1802.08312/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2018.

## Built on by (most cited first, 11 of 11)

- [Benchmarking LLMs' Judgments with No Gold Standard](https://mlsanity.com/paper/2411.07127/) (2024)
- [Eliciting Informative Text Evaluations with Large Language Models](https://mlsanity.com/paper/2405.15077/) (2024)
- [ElicitationGPT: Text Elicitation Mechanisms via Language Models](https://mlsanity.com/paper/2406.09363/) (2024)
- [SelectMix: Enhancing Label Noise Robustness through Targeted Sample Mixing](https://mlsanity.com/paper/2509.11265/) (2025)
- [Spot Check Equivalence: an Interpretable Metric for Information Elicitation Mechanisms](https://mlsanity.com/paper/2402.13567/) (2024)
- [Auctions and Peer Prediction for Academic Peer Review](https://mlsanity.com/paper/2109.00923/) (2021)
- [Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth](https://mlsanity.com/paper/2506.06991/) (2025)
- [Surprisingly Popular Voting for Concentric Rank-Order Models](https://mlsanity.com/paper/2411.08367/) (2024)
- [Improving Cryptocurrency Pump-and-Dump Detection through Ensemble-Based Models and Synthetic Oversampling Techniques](https://mlsanity.com/paper/2510.00836/) (2025)
- [SP-Rank: A Dataset for Ranked Preferences with Secondary Information](https://mlsanity.com/paper/2601.05253/) (2025)
- [Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities](https://mlsanity.com/paper/2605.10810/) (2026)

## Abstract (arXiv)

A central question of crowd-sourcing is how to elicit expertise from agents. This is even more difficult when answers cannot be directly verified. A key challenge is that sophisticated agents may strategically withhold effort or information when they believe their payoff will be based upon comparison with other agents whose reports will likely omit this information due to lack of effort or expertise. Our work defines a natural model for this setting based on the assumption that more sophisticated agents know the beliefs of less sophisticated agents. We then provide a mechanism design framework for this setting. From this framework, we design several novel mechanisms, for both the single and multiple question settings, that (1) encourage agents to invest effort and provide their information honestly; (2) output a correct "hierarchy" of the information when agents are rational.