# Crowdsourced Judgement Elicitation with Endogenous Proficiency

arXiv: https://arxiv.org/abs/1303.0799 · HTML: https://mlsanity.com/paper/1303.0799/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2013.

## Built on by (most cited first, 30 of 30)

- [Benchmarking LLMs' Judgments with No Gold Standard](https://mlsanity.com/paper/2411.07127/) (2024)
- [Data Pricing in Machine Learning Pipelines](https://mlsanity.com/paper/2108.07915/) (2021)
- [Eliciting Informative Text Evaluations with Large Language Models](https://mlsanity.com/paper/2405.15077/) (2024)
- [Data-Free Evaluation of User Contributions in Federated Learning](https://mlsanity.com/paper/2108.10623/) (2021)
- [Learning from Pixel-Level Noisy Label : A New Perspective for Light Field Saliency Detection](https://mlsanity.com/paper/2204.13456/) (2022)
- [SelectMix: Enhancing Label Noise Robustness through Targeted Sample Mixing](https://mlsanity.com/paper/2509.11265/) (2025)
- [Surprisingly Popular Voting Recovers Rankings, Surprisingly!](https://mlsanity.com/paper/2105.09386/) (2021)
- [Truthful Incentive Mechanism for Federated Learning with Crowdsourced Data Labeling](https://mlsanity.com/paper/2302.00106/) (2023)
- [Spot Check Equivalence: an Interpretable Metric for Information Elicitation Mechanisms](https://mlsanity.com/paper/2402.13567/) (2024)
- [Reward-Based 1-bit Compressed Federated Distillation on Blockchain](https://mlsanity.com/paper/2106.14265/) (2021)
- [Auctions and Peer Prediction for Academic Peer Review](https://mlsanity.com/paper/2109.00923/) (2021)
- [Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth](https://mlsanity.com/paper/2506.06991/) (2025)
- [Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores](https://mlsanity.com/paper/2510.14966/) (2025)
- [Data Reliability Scoring](https://mlsanity.com/paper/2510.17085/) (2025)
- [Truthful Generalized Linear Models](https://mlsanity.com/paper/2209.07815/) (2022)
- [On The Truthfulness of 'Surprisingly Likely' Responses of Large Language Models](https://mlsanity.com/paper/2311.07692/) (2023)
- [Navigating Towards Fairness with Data Selection](https://mlsanity.com/paper/2412.11072/) (2024)
- [Stochastically Dominant Peer Prediction](https://mlsanity.com/paper/2506.02259/) (2025)
- [Incentive-Aligned Multi-Source LLM Summaries](https://mlsanity.com/paper/2509.25184/) (2025)
- [Incentivizing an Unknown Crowd](https://mlsanity.com/paper/2109.04226/) (2021)
- [Information Elicitation Meets Clustering](https://mlsanity.com/paper/2110.00952/) (2021)
- [Improving Peer Assessment with Graph Convolutional Networks](https://mlsanity.com/paper/2111.04466/) (2021)
- [Bayes classifier cannot be learned from noisy responses with unknown noise rates](https://mlsanity.com/paper/2304.06574/) (2023)
- [Incentive-Compatible Recovery from Manipulated Signals, with Applications to Decentralized Physical Infrastructure](https://mlsanity.com/paper/2503.07558/) (2025)
- [Incentivizing High-Quality Human Annotations with Golden Questions](https://mlsanity.com/paper/2505.19134/) (2025)
- [A Cram'er-von Mises Approach to Incentivizing Truthful Data Sharing](https://mlsanity.com/paper/2506.07272/) (2025)
- [Users as Annotators: LLM Preference Learning from Comparison Mode](https://mlsanity.com/paper/2510.13830/) (2025)
- [Multi-agent Adaptive Mechanism Design](https://mlsanity.com/paper/2512.21794/) (2025)
- [Knowledge-Free Correlated Agreement for Incentivizing Federated Learning](https://mlsanity.com/paper/2605.04747/) (2026)
- [Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics](https://mlsanity.com/paper/2608.01423/) (2026)

## Abstract (arXiv)

Crowdsourcing is now widely used to replace judgement by an expert authority with an aggregate evaluation from a number of non-experts, in applications ranging from rating and categorizing online content to evaluation of student assignments in massively open online courses via peer grading. A key issue in these settings, where direct monitoring is infeasible, is incentivizing agents in the `crowd' to put in effort to make good evaluations, as well as to truthfully report their evaluations. This leads to a new family of information elicitation problems with unobservable ground truth, where an agent's proficiency- the probability with which she correctly evaluates the underlying ground truth- is endogenously determined by her strategic choice of how much effort to put into the task. Our main contribution is a simple, new, mechanism for binary information elicitation for multiple tasks when agents have endogenous proficiencies, with the following properties: (i) Exerting maximum effort followed by truthful reporting of observations is a Nash equilibrium. (ii) This is the equilibrium with maximum payoff to all agents, even when agents have different maximum proficiencies, can use mixed strategies, and can choose a different strategy for each of their tasks. Our information elicitation mechanism requires only minimal bounds on the priors, asks agents to only report their own evaluations, and does not require any conditions on a diverging number of agent reports per task to achieve its incentive properties. The main idea behind our mechanism is to use the presence of multiple tasks and ratings to identify and penalize low-effort agreement: the mechanism rewards agents for agreeing with a `reference' rater on a task but also penalizes for blind agreement by subtracting out a statistic term designed so that agents obtain reward only when they put effort into their observations.