# Principled Multi-Aspect Evaluation Measures of Rankings

arXiv: https://arxiv.org/abs/2212.00492 · HTML: https://mlsanity.com/paper/2212.00492/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2022.

## Built on by (most cited first, 3 of 3)

- [Stairway to Fairness: Connecting Group and Individual Fairness](https://mlsanity.com/paper/2508.21334/) (2025)
- [Measuring Individual User Fairness with User Similarity and Effectiveness Disparity](https://mlsanity.com/paper/2602.02516/) (2026)
- [Advanced Acceptance Score: A Holistic Measure for Biometric Quantification](https://mlsanity.com/paper/2602.15535/) (2026)

## Abstract (arXiv)

Information Retrieval evaluation has traditionally focused on defining principled ways of assessing the relevance of a ranked list of documents with respect to a query. Several methods extend this type of evaluation beyond relevance, making it possible to evaluate different aspects of a document ranking (e.g., relevance, usefulness, or credibility) using a single measure (multi-aspect evaluation). However, these methods either are (i) tailor-made for specific aspects and do not extend to other types or numbers of aspects, or (ii) have theoretical anomalies, e.g. assign maximum score to a ranking where all documents are labelled with the lowest grade with respect to all aspects (e.g., not relevant, not credible, etc.). We present a theoretically principled multi-aspect evaluation method that can be used for any number, and any type, of aspects. A thorough empirical evaluation using up to 5 aspects and a total of 425 runs officially submitted to 10 TREC tracks shows that our method is more discriminative than the state-of-the-art and overcomes theoretical limitations of the state-of-the-art.