# Modeling and Correcting Bias in Sequential Evaluation

arXiv: https://arxiv.org/abs/2205.01607 · HTML: https://mlsanity.com/paper/2205.01607/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 62 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 11 of 11)

- [Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence](https://mlsanity.com/paper/1505.01462/) (2015)
- [Design and Analysis of the NIPS 2016 Review Process](https://mlsanity.com/paper/1708.09794/) (2017)
- [Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issues](https://mlsanity.com/paper/1510.05610/) (2015)
- [Linear regression without correspondence](https://mlsanity.com/paper/1705.07048/) (2017)
- [A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness](https://mlsanity.com/paper/1606.09632/) (2016)
- [Isotonic regression with unknown permutations: Statistics, computation, and adaptation](https://mlsanity.com/paper/2009.02609/) (2020)
- [Denoising Linear Models with Permuted Data](https://mlsanity.com/paper/1704.07461/) (2017)
- [Optimal Rates of Statistical Seriation](https://mlsanity.com/paper/1607.02435/) (2016)
- [Debiasing Evaluations That are Biased by Evaluations](https://mlsanity.com/paper/2012.00714/) (2020)
- [Minimax Rates in Network Analysis: Graphon Estimation, Community Detection and Hypothesis Testing](https://mlsanity.com/paper/1811.06055/) (2018)
- [Designing Optimal Binary Rating Systems](https://mlsanity.com/paper/1806.06908/) (2018)

## Abstract (arXiv)

We consider the problem of sequential evaluation, in which an evaluator observes candidates in a sequence and assigns scores to these candidates in an online, irrevocable fashion. Motivated by the psychology literature that has studied sequential bias in such settings -- namely, dependencies between the evaluation outcome and the order in which the candidates appear -- we propose a natural model for the evaluator's rating process that captures the lack of calibration inherent to such a task. We conduct crowdsourcing experiments to demonstrate various facets of our model. We then proceed to study how to correct sequential bias under our model by posing this as a statistical inference problem. We propose a near-linear time, online algorithm for this task and prove guarantees in terms of two canonical ranking metrics. We also prove that our algorithm is information theoretically optimal, by establishing matching lower bounds in both metrics. Finally, we perform a host of numerical experiments to show that our algorithm often outperforms the de facto method of using the rankings induced by the reported scores, both in simulation and on the crowdsourcing data that we collected.