# SlovKE: A Large-Scale Dataset and LLM Evaluation for Slovak Keyphrase Extraction

arXiv: https://arxiv.org/abs/2603.15523 · HTML: https://mlsanity.com/paper/2603.15523/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Deep Keyphrase Generation](https://mlsanity.com/paper/1704.06879/) (2017)
- [KPTimes: A Large-Scale Dataset for Keyphrase Generation on News Documents](https://mlsanity.com/paper/1911.12559/) (2019)
- [A Review of Keyphrase Extraction](https://mlsanity.com/paper/1905.05044/) (2019)
- [Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training](https://mlsanity.com/paper/2205.10471/) (2022)
- [Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study](https://mlsanity.com/paper/2212.10233/) (2022)
- [EUROPA: A Legal Multilingual Keyphrase Generation Dataset](https://mlsanity.com/paper/2403.00252/) (2024)
- [Query-Based Keyphrase Extraction from Long Documents](https://mlsanity.com/paper/2205.05391/) (2022)
- [Keyword Extraction from Short Texts with a Text-To-Text Transfer Transformer](https://mlsanity.com/paper/2209.14008/) (2022)

## Abstract (arXiv)

Keyphrase extraction for morphologically rich, low-resource languages remains understudied, largely due to the scarcity of suitable evaluation datasets. We address this gap for Slovak by constructing a dataset of 227,432 scientific abstracts with author-assigned keyphrases -- scraped and systematically cleaned from the Slovak Central Register of Theses -- representing a 25-fold increase over the largest prior Slovak resource and approaching the scale of established English benchmarks such as KP20K. Using this dataset, we benchmark three unsupervised baselines (YAKE, TextRank, KeyBERT with SlovakBERT embeddings) and evaluate KeyLLM, an LLM-based extraction method using GPT-3.5-turbo. Unsupervised baselines achieve at most 11.6% exact-match F1@6, with a large gap to partial matching (up to 51.5%), reflecting the difficulty of matching inflected surface forms to author-assigned keyphrases. KeyLLM narrows this exact--partial gap, producing keyphrases closer to the canonical forms assigned by authors, while manual evaluation on 100 documents (kappa = 0.61) confirms that KeyLLM captures relevant concepts that automated exact matching underestimates. Our analysis identifies morphological mismatch as the dominant failure mode for statistical methods -- a finding relevant to other inflected languages. The dataset (https://huggingface.co/datasets/NaiveNeuron/SlovKE) and evaluation code (https://github.com/NaiveNeuron/SlovKE) are publicly available.