Uncertainty in Ranking

Ranks estimated from data are uncertain and this poses a challenge in many applications. However, estimated ranks are deterministic functions of estimated parameters, so the uncertainty in the ranks must be determined by the uncertainty in the parameter estimates. We give a complete characterization of this relationship in terms of the linear extensions of a partial order determined by interval estimates of the parameters of interest. We then use this relationship to give a set estimator for the overall ranking, use its size to measure the uncertainty in a ranking, and give efficient algorithms for several questions of interest. We show that our set estimator is a valid confidence set and describe its relationship to a joint confidence set for ranks recently proposed by Klein, Wright & Wieczorek. We apply our methods to both simulated and real data and make them available through the R package rankUncertainty.