# Overview of the TREC 2019 Fair Ranking Track

arXiv: https://arxiv.org/abs/2003.11650 · HTML: https://mlsanity.com/paper/2003.11650/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2020.

## Built on by (most cited first, 4 of 4)

- [Estimation of Fair Ranking Metrics with Incomplete Judgments](https://mlsanity.com/paper/2108.05152/) (2021)
- [Fairness Through Regularization for Learning to Rank](https://mlsanity.com/paper/2102.05996/) (2021)
- [Fairness in Image Search: A Study of Occupational Stereotyping in Image Retrieval and its Debiasing](https://mlsanity.com/paper/2305.03881/) (2023)
- [Quantifying Misattribution Unfairness in Authorship Attribution](https://mlsanity.com/paper/2506.02321/) (2025)

## Abstract (arXiv)

The goal of the TREC Fair Ranking track was to develop a benchmark for evaluating retrieval systems in terms of fairness to different content providers in addition to classic notions of relevance. As part of the benchmark, we defined standardized fairness metrics with evaluation protocols and released a dataset for the fair ranking problem. The 2019 task focused on reranking academic paper abstracts given a query. The objective was to fairly represent relevant authors from several groups that were unknown at the system submission time. Thus, the track emphasized the development of systems which have robust performance across a variety of group definitions. Participants were provided with querylog data (queries, documents, and relevance) from Semantic Scholar. This paper presents an overview of the track, including the task definition, descriptions of the data and the annotation process, as well as a comparison of the performance of submitted systems.