mlsanityData to 14 Oct 2026

SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

arXiv 2606.13647 ↗First posted 11 Jun 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types -- nearly 4× the depth of existing multilingual benchmark coverage for Slovak. Our evaluation of 31 embedding models reveals that large instruction-tuned multilingual models achieve the strongest performance, while existing Slovak-specific models trained for NLU tasks transfer poorly to embedding tasks. To address the need for efficient, locally-deployable Slovak embeddings, we develop texttte5-sk-small (45M parameters) and texttte5-sk-large (365M) by applying vocabulary trimming and fine-tuning to Multilingual E5 models. Despite size reductions of up to 62%, our open-source models achieve competitive performance with proprietary APIs while remaining locally deployable for semantic search and retrieval-augmented generation (RAG). We release the benchmark, models, datasets, and code openly, hoping our approach offers a replicable path for other under-resourced languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

37 of 62 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  6. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  9. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  10. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  11. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
25 more
  1. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  2. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  3. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
  4. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  5. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 229
  6. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  7. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  8. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 141
  9. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
  10. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 125
  11. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
  12. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 76
  13. 2024Arctic-Embed 2.0: Multilingual Retrieval Without Compromisecited by 58
  14. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  15. 2024The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embeddingcited by 26
  16. 2023An Efficient Multilingual Language Model Compression through Vocabulary Trimmingcited by 20
  17. 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
  18. 2024PL-MTEB: Polish Massive Text Embedding Benchmarkcited by 19
  19. 2021MFAQ: a Multilingual FAQ Datasetcited by 17
  20. 2024Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarkscited by 10
  21. 2025FaMTEB: Massive Text Embedding Benchmark in Persian Languagecited by 10
  22. 2025WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrievalcited by 10
  23. 2023Resources and Few-shot Learners for In-context Learning in Slavic Languagescited by 6
  24. 2023The ParlaSent Multilingual Training Dataset for Sentiment Identification in Parliamentary Proceedingscited by 6
  25. 2025skLEP: A Slovak General Language Understanding Benchmarkcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak
  2. 2025MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
  3. 2026Less is More: Adapting Text Embeddings for Low-Resource Languages with Small Scale Noisy Synthetic Data
  4. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark
  5. 2025KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model
  6. 2023Some Like It Small: Czech Semantic Embedding Models for Industry Applications
  7. 2025Hakim: Farsi Text Embedding Model
  8. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World