# The Quest for Reliable Metrics of Responsible AI

arXiv: https://arxiv.org/abs/2510.26007 · HTML: https://mlsanity.com/paper/2510.26007/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 8 of 8)

- [A Survey on the Fairness of Recommender Systems](https://mlsanity.com/paper/2206.03761/) (2022)
- [A Survey on Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2308.01118/) (2023)
- [Citation Recommendation: Approaches and Datasets](https://mlsanity.com/paper/2002.06961/) (2020)
- [Scientific Paper Recommendation Systems: a Literature Review of recent Publications](https://mlsanity.com/paper/2201.00682/) (2022)
- [Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study](https://mlsanity.com/paper/2311.01013/) (2023)
- [Overview of the TREC 2020 Fair Ranking Track](https://mlsanity.com/paper/2108.05135/) (2021)
- [Stairway to Fairness: Connecting Group and Individual Fairness](https://mlsanity.com/paper/2508.21334/) (2025)
- [Joint Evaluation of Fairness and Relevance in Recommender Systems with Pareto Frontier](https://mlsanity.com/paper/2502.11921/) (2025)

## Abstract (arXiv)

The development of Artificial Intelligence (AI), including AI in Science (AIS), should be done following the principles of responsible AI. Progress in responsible AI is often quantified through evaluation metrics, yet there has been less work on assessing the robustness and reliability of the metrics themselves. We reflect on prior work that examines the robustness of fairness metrics for recommender systems as a type of AI application and summarise their key takeaways into a set of non-exhaustive guidelines for developing reliable metrics of responsible AI. Our guidelines apply to a broad spectrum of AI applications, including AIS.