# A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevance

arXiv: https://arxiv.org/abs/2204.00280 · HTML: https://mlsanity.com/paper/2204.00280/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2022.

## Built on by (most cited first, 4 of 4)

- [Measuring and Addressing Indexical Bias in Information Retrieval](https://mlsanity.com/paper/2406.04298/) (2024)
- [SWAN: A Generic Framework for Auditing Textual Conversational Systems](https://mlsanity.com/paper/2305.08290/) (2023)
- [Advancing Re-Ranking with Multimodal Fusion and Target-Oriented Auxiliary Tasks in E-Commerce Search](https://mlsanity.com/paper/2408.05751/) (2024)
- [Plans for Evaluating Structured Generative Search Summaries](https://mlsanity.com/paper/2605.26400/) (2026)

## Abstract (arXiv)

We present a simple and versatile framework for evaluating ranked lists in terms of group fairness and relevance, where the groups (i.e., possible attribute values) can be either nominal or ordinal in nature. First, we demonstrate that, if the attribute set is binary, our framework can easily quantify the overall polarity of each ranked list. Second, by utilising an existing diversified search test collection and treating each intent as an attribute value, we demonstrate that our framework can handle soft group membership, and that our group fairness measures are highly correlated with both adhoc IR and diversified IR measures under this setting. Third, we demonstrate how our framework can quantify intersectional group fairness based on multiple attribute sets. We also show that the similarity function for comparing the achieved and target distributions over the attribute values should be chosen carefully.