mlsanityData to 14 Oct 2026

skLEP: A Slovak General Language Understanding Benchmark

arXiv 2506.21508 ↗First posted 26 Jun 2025cs.CL cs.AI cs.IRcited by 3

Not on the map. Why?

Abstract (arXiv)

In this work, we introduce skLEP, the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding (NLU) models. We have compiled skLEP to encompass nine diverse tasks that span token-level, sentence-pair, and document-level challenges, thereby offering a thorough assessment of model capabilities. To create this benchmark, we curated new, original datasets tailored for Slovak and meticulously translated established English NLU resources. Within this paper, we also present the first systematic and extensive evaluation of a wide array of Slovak-specific, multilingual, and English pre-trained language models using the skLEP tasks. Finally, we also release the complete benchmark data, an open-source toolkit facilitating both fine-tuning and evaluation of models, and a public leaderboard at https://github.com/slovak-nlp/sklep in the hopes of fostering reproducibility and drive future research in Slovak NLU.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 49 of its 72 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

49 of 72 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  10. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
37 more
  1. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  4. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  5. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  6. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  7. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  8. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  9. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  10. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  11. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  12. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  13. 2021KLUE: Korean Language Understanding Evaluationcited by 114
  14. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
  15. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  16. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  17. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  18. 2021Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarkingcited by 53
  19. 2020KLEJ: Comprehensive Benchmark for Polish Language Understandingcited by 51
  20. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 47
  21. 2020ParsiNLU: A Suite of Language Understanding Challenges for Persiancited by 42
  22. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  23. 2019Post-editese: an Exacerbated Translationesecited by 29
  24. 2023Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmarkcited by 26
  25. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  26. 2022Dynatask: A Framework for Creating Dynamic AI Benchmark Taskscited by 16
  27. 2022This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polishcited by 13
  28. 2021SlovakBERT: Slovak Masked Language Modelcited by 11
  29. 2021Comparison of Czech Transformers on Text Classification Taskscited by 10
  30. 2022Slovene SuperGLUE Benchmark: Translation and Evaluationcited by 7
  31. 2024Large Language Models "Ad Referendum": How Good Are They at Machine Translation in the Legal Domain?cited by 7
  32. 2024GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledgecited by 7
  33. 2023DUMB: A Benchmark for Smart Evaluation of Dutch Modelscited by 6
  34. 2024PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguesecited by 5
  35. 2023bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmarkcited by 4
  36. 2024The Invalsi Benchmarks: measuring Linguistic and Mathematical understanding of Large Language Models in Italiancited by 4
  37. 2021The Catalan Language CLUBcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmarkcited by 0
  2. 2026SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptationcited by 0
  3. 2026Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languagescited by 0