# A Weighted Correlation Index for Rankings with Ties

arXiv: https://arxiv.org/abs/1404.3325 · HTML: https://mlsanity.com/paper/1404.3325/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2014.

## Built on by (most cited first, 30 of 41)

- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs](https://mlsanity.com/paper/2110.10545/) (2021)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [SurvSHAP(t): Time-dependent explanations of machine learning survival models](https://mlsanity.com/paper/2208.11080/) (2022)
- [Which Model to Transfer? A Survey on Transferability Estimation](https://mlsanity.com/paper/2402.15231/) (2024)
- [OpBoost: A Vertical Federated Tree Boosting Framework Based on Order-Preserving Desensitization](https://mlsanity.com/paper/2210.01318/) (2022)
- [ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization](https://mlsanity.com/paper/2210.09236/) (2022)
- [A Bag-of-Prototypes Representation for Dataset-Level Applications](https://mlsanity.com/paper/2303.13251/) (2023)
- [Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers](https://mlsanity.com/paper/2412.06639/) (2024)
- [Probing for Labeled Dependency Trees](https://mlsanity.com/paper/2203.12971/) (2022)
- [Average Localised Proximity: A new data descriptor with good default one-class classification performance](https://mlsanity.com/paper/2101.11037/) (2021)
- [Evidence > Intuition: Transferability Estimation for Encoder Selection](https://mlsanity.com/paper/2210.11255/) (2022)
- [Evaluating AI systems under uncertain ground truth: a case study in dermatology](https://mlsanity.com/paper/2307.02191/) (2023)
- [Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach](https://mlsanity.com/paper/2309.02429/) (2023)
- [RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank](https://mlsanity.com/paper/2405.01848/) (2024)
- [Optimised one-class classification performance](https://mlsanity.com/paper/2102.02618/) (2021)
- [On genetic programming representations and fitness functions for interpretable dimensionality reduction](https://mlsanity.com/paper/2203.00528/) (2022)
- [Instability in clinical risk stratification models using deep learning](https://mlsanity.com/paper/2211.10828/) (2022)
- [Metric Learning Encoding Models: A Multivariate Framework for Interpreting Neural Representations](https://mlsanity.com/paper/2402.11608/) (2024)
- [I Bet You Did Not Mean That: Testing Semantic Importance via Betting](https://mlsanity.com/paper/2405.19146/) (2024)
- [Sort by Structure: Language Model Ranking as Dependency Probing](https://mlsanity.com/paper/2206.04935/) (2022)
- [Benchmarking Transferability: A Framework for Fair and Robust Evaluation](https://mlsanity.com/paper/2504.20121/) (2025)
- [How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluation](https://mlsanity.com/paper/2510.06448/) (2025)
- [KITE: A Kernel-based Improved Transferability Estimation Method](https://mlsanity.com/paper/2405.01603/) (2024)
- [What Makes Two Language Models Think Alike?](https://mlsanity.com/paper/2406.12620/) (2024)
- [CoxSE: Exploring the Potential of Self-Explaining Neural Networks with Cox Proportional Hazards Model for Survival Analysis](https://mlsanity.com/paper/2407.13849/) (2024)
- [k-NN as a Simple and Effective Estimator of Transferability](https://mlsanity.com/paper/2503.18528/) (2025)
- [Standardization of Weighted Ranking Correlation Coefficients](https://mlsanity.com/paper/2504.08428/) (2025)
- [Analysis of Transferability Estimation Metrics for Surgical Phase Recognition](https://mlsanity.com/paper/2508.16730/) (2025)

## Abstract (arXiv)

Understanding the correlation between two different scores for the same set of items is a common problem in information retrieval, and the most commonly used statistics that quantifies this correlation is Kendall's τ. However, the standard definition fails to capture that discordances between items with high rank are more important than those between items with low rank. Recently, a new measure of correlation based on average precision has been proposed to solve this problem, but like many alternative proposals in the literature it assumes that there are no ties in the scores. This is a major deficiency in a number of contexts, and in particular while comparing centrality scores on large graphs, as the obvious baseline, indegree, has a very large number of ties in web and social graphs. We propose to extend Kendall's definition in a natural way to take into account weights in the presence of ties. We prove a number of interesting mathematical properties of our generalization and describe an O(nlog n) algorithm for its computation. We also validate the usefulness of our weighted measure of correlation using experimental data.