# Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models

arXiv: https://arxiv.org/abs/2010.04486 · HTML: https://mlsanity.com/paper/2010.04486/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2020.

## Built on by (most cited first, 4 of 4)

- [Standardization of Weighted Ranking Correlation Coefficients](https://mlsanity.com/paper/2504.08428/) (2025)
- [Towards Explainable Job Title Matching: Leveraging Semantic Textual Relatedness and Knowledge Graphs](https://mlsanity.com/paper/2509.09522/) (2025)
- [Cost-Sensitive Evaluation for Binary Classifiers](https://mlsanity.com/paper/2510.22016/) (2025)
- [Matching Ranks Over Probability Yields Truly Deep Safety Alignment](https://mlsanity.com/paper/2512.05518/) (2025)

## Abstract (arXiv)

The growth of domain-specific applications of semantic models, boosted by the recent achievements of unsupervised embedding learning algorithms, demands domain-specific evaluation datasets. In many cases, content-based recommenders being a prime example, these models are required to rank words or texts according to their semantic relatedness to a given concept, with particular focus on top ranks. In this work, we give a threefold contribution to address these requirements: (i) we define a protocol for the construction, based on adaptive pairwise comparisons, of a relatedness-based evaluation dataset tailored on the available resources and optimized to be particularly accurate in top-rank evaluation; (ii) we define appropriate metrics, extensions of well-known ranking correlation coefficients, to evaluate a semantic model via the aforementioned dataset by taking into account the greater significance of top ranks. Finally, (iii) we define a stochastic transitivity model to simulate semantic-driven pairwise comparisons, which confirms the effectiveness of the proposed dataset construction protocol.