# Optimal Scoring Rule Design under Partial Knowledge

arXiv: https://arxiv.org/abs/2107.07420 · HTML: https://mlsanity.com/paper/2107.07420/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 17 of 17)

- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Strategic Classification](https://mlsanity.com/paper/1506.06980/) (2015)
- [A Marketplace for Data: An Algorithmic Solution](https://mlsanity.com/paper/1805.08125/) (2018)
- [Optimum Statistical Estimation with Strategic Data Sources](https://mlsanity.com/paper/1408.2539/) (2014)
- [Optimization of Scoring Rules](https://mlsanity.com/paper/2007.02905/) (2020)
- [Strategyproof Linear Regression in High Dimensions](https://mlsanity.com/paper/1805.10693/) (2018)
- [Contracts under Moral Hazard and Adverse Selection](https://mlsanity.com/paper/2010.06742/) (2020)
- [Truthful Linear Regression](https://mlsanity.com/paper/1506.03489/) (2015)
- [Binary Scoring Rules that Incentivize Precision](https://mlsanity.com/paper/2002.10669/) (2020)
- [Redrawing the Boundaries on Purchasing Data from Privacy-Sensitive Individuals](https://mlsanity.com/paper/1401.4092/) (2014)
- [Optimal Data Acquisition for Statistical Estimation](https://mlsanity.com/paper/1711.01295/) (2017)
- [Buying Private Data without Verification](https://mlsanity.com/paper/1404.6003/) (2014)
- [Prior-free Data Acquisition for Accurate Statistical Estimation](https://mlsanity.com/paper/1811.12655/) (2018)
- [Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication](https://mlsanity.com/paper/2005.04543/) (2020)
- [Active Information Acquisition for Linear Optimization](https://mlsanity.com/paper/1709.10061/) (2017)
- [Optimal Auctions for Correlated Buyers with Sampling](https://mlsanity.com/paper/1406.1571/) (2014)
- [Delegated Stochastic Probing](https://mlsanity.com/paper/2010.14718/) (2020)

## Built on by (most cited first, 4 of 4)

- [Multi-agent Performative Prediction: From Global Stability and Optimality to Chaos](https://mlsanity.com/paper/2201.10483/) (2022)
- [Truthfulness of Calibration Measures](https://mlsanity.com/paper/2407.13979/) (2024)
- [Choice of Scoring Rules for Indirect Elicitation of Properties with Parametric Assumptions](https://mlsanity.com/paper/2506.17880/) (2025)
- [Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics](https://mlsanity.com/paper/2608.01423/) (2026)

## Abstract (arXiv)

This paper studies the design of optimal proper scoring rules when the principal has partial knowledge of an agent's signal distribution. Recent work characterizes the proper scoring rules that maximize the increase of an agent's payoff when the agent chooses to access a costly signal to refine a posterior belief from her prior prediction, under the assumption that the agent's signal distribution is fully known to the principal. In our setting, the principal only knows about a set of distributions where the agent's signal distribution belongs. We formulate the scoring rule design problem as a max-min optimization that maximizes the worst-case increase in payoff across the set of distributions. We propose an efficient algorithm to compute an optimal scoring rule when the set of distributions is finite, and devise a fully polynomial-time approximation scheme that accommodates various infinite sets of distributions. We further remark that widely used scoring rules, such as the quadratic and log rules, as well as previously identified optimal scoring rules under full knowledge, can be far from optimal in our partial knowledge settings.