# Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation

arXiv: https://arxiv.org/abs/2606.07057 · HTML: https://mlsanity.com/paper/2606.07057/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Builds on (most cited first, 12 of 12)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Deep Keyphrase Generation](https://mlsanity.com/paper/1704.06879/) (2017)
- [One Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases](https://mlsanity.com/paper/1810.05241/) (2018)
- [Unsupervised Keyphrase Extraction with Multipartite Graphs](https://mlsanity.com/paper/1803.08721/) (2018)
- [KPTimes: A Large-Scale Dataset for Keyphrase Generation on News Documents](https://mlsanity.com/paper/1911.12559/) (2019)
- [One2Set: Generating Diverse Keyphrases as a Set](https://mlsanity.com/paper/2105.11134/) (2021)
- [Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study](https://mlsanity.com/paper/2212.10233/) (2022)
- [Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective](https://mlsanity.com/paper/2411.14654/) (2024)
- [Hyperbolic Relevance Matching for Neural Keyphrase Extraction](https://mlsanity.com/paper/2205.02047/) (2022)
- [KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation](https://mlsanity.com/paper/2303.15422/) (2023)
- [KOIOS: Top-k Semantic Overlap Set Search](https://mlsanity.com/paper/2304.10572/) (2023)

## Abstract (arXiv)

Evaluating the quality of automatically generated keyphrases remains a complex challenge. Traditional metrics either rely on exact lexical matching or consider semantic similarity while ignoring prediction ranking, both of which misalign with how humans judge informativeness and relevance. We introduce Semantic R-Precision (SemR-p), a novel evaluation metric that integrates semantic similarity into the rank-aware R-Precision framework. Designed from a human-centric perspective and inspired by Information Retrieval metrics, SemR-p rewards semantically relevant keyphrases that appear early in the output list. We conducted extensive analyses to assess its semantic sensitivity, ranking awareness, and discriminative power across models and datasets. The results suggest that SemR-p offers a complementary lens for evaluating keyphrase predictions, helping to better reflect user-centred notions of relevance alongside traditional lexical and semantic matching metrics.