# Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?

arXiv: https://arxiv.org/abs/2206.12858 · HTML: https://mlsanity.com/paper/2206.12858/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2022.

## Built on by (most cited first, 17 of 17)

- [The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs](https://mlsanity.com/paper/2303.12961/) (2023)
- [Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations](https://mlsanity.com/paper/2408.12008/) (2024)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Efficient Discovery and Effective Evaluation of Visual Perceptual Similarity: A Benchmark and Beyond](https://mlsanity.com/paper/2308.14753/) (2023)
- [Building a Scalable, Effective, and Steerable Search and Ranking Platform](https://mlsanity.com/paper/2409.02856/) (2024)
- [RePlay: a Recommendation Framework for Experimentation and Production Use](https://mlsanity.com/paper/2409.07272/) (2024)
- [Knowledge Graph Completion with Mixed Geometry Tensor Factorization](https://mlsanity.com/paper/2504.02589/) (2025)
- [Reusable Self-Attention-based Recommender System for Fashion](https://mlsanity.com/paper/2211.16366/) (2022)
- [Reducing Popularity Influence by Addressing Position Bias](https://mlsanity.com/paper/2412.08780/) (2024)
- [Generative Representational Learning of Foundation Models for Recommendation](https://mlsanity.com/paper/2506.11999/) (2025)
- [Are We Really Measuring Progress? Transferring Insights from Evaluating Recommender Systems to Temporal Link Prediction](https://mlsanity.com/paper/2506.12588/) (2025)
- [Efficient and Effective Query Context-Aware Learning-to-Rank Model for Sequential Recommendation](https://mlsanity.com/paper/2507.03789/) (2025)
- [Where Do We Go From Here? Guidelines For Offline Recommender Evaluation](https://mlsanity.com/paper/2211.01261/) (2022)
- [Decision-Aligned Evaluation of Uncertainty Quantification](https://mlsanity.com/paper/2606.26990/) (2026)
- [Job recommendations: benchmarking of collaborative filtering methods for classifieds](https://mlsanity.com/paper/2301.07946/) (2023)
- [Supplier Recommendation in Online Procurement](https://mlsanity.com/paper/2403.01301/) (2024)
- [Beyond Factual Correctness: Mitigating Preference-Inconsistent Explanations in Explainable Recommendation](https://mlsanity.com/paper/2603.03080/) (2026)

## Abstract (arXiv)

Offline evaluation is a popular approach to determine the best algorithm in terms of the chosen quality metric. However, if the chosen metric calculates something unexpected, this miscommunication can lead to poor decisions and wrong conclusions. In this paper, we thoroughly investigate quality metrics used for recommender systems evaluation. We look at the practical aspect of implementations found in modern RecSys libraries and at the theoretical aspect of definitions in academic papers. We find that Precision is the only metric universally understood among papers and libraries, while other metrics may have different interpretations. Metrics implemented in different libraries sometimes have the same name but measure different things, which leads to different results given the same input. When defining metrics in an academic paper, authors sometimes omit explicit formulations or give references that do not contain explanations either. In 47% of cases, we cannot easily know how the metric is defined because the definition is not clear or absent. These findings highlight yet another difficulty in recommender system evaluation and call for a more detailed description of evaluation protocols.