mlsanityData to 14 Oct 2026

The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding

arXiv 2406.02396 ↗First posted 4 Jun 2024cs.CL cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

The evaluation of English text embeddings has transitioned from evaluating a handful of datasets to broad coverage across many tasks through benchmarks such as MTEB. However, this is not the case for multilingual text embeddings due to a lack of available benchmarks. To address this problem, we introduce the Scandinavian Embedding Benchmark (SEB). SEB is a comprehensive framework that enables text embedding evaluation for Scandinavian languages across 24 tasks, 10 subtasks, and 4 task categories. Building on SEB, we evaluate more than 26 models, uncovering significant performance disparities between public and commercial solutions not previously captured by MTEB. We open-source SEB and integrate it with MTEB, thus bridging the text embedding evaluation gap for Scandinavian languages.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 36 of its 70 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

36 of 70 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  11. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  12. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
24 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  2. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  3. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  4. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  5. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  7. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  8. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  9. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  10. 2016FastText.zip: Compressing text classification modelscited by 335
  11. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  12. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  13. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  14. 2024Generative Representational Instruction Tuningcited by 233
  15. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  16. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  17. 2020Top2Vec: Distributed Representations of Topicscited by 126
  18. 2019Offensive Language and Hate Speech Detection for Danishcited by 49
  19. 2021Operationalizing a National Digital Library: The Case for a Norwegian Transformer Modelcited by 23
  20. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  21. 2017NoReC: The Norwegian Review Corpuscited by 17
  22. 2020The Danish Gigaword Projectcited by 17
  23. 2020Discriminating Between Similar Nordic Languagescited by 10
  24. 2023Danish Foundation Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 10; 2026: 8.

26 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 36
  3. 2024Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Taskscited by 34
  4. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  5. 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
  6. 2025MIEB: Massive Image Embedding Benchmarkcited by 15
  7. 2024Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and Cross-Cultural Embedding Models and Benchmarkscited by 10
  8. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  9. 2025NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmarkcited by 8
  10. 2025Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarkscited by 8
  11. 2025HUME: Measuring the Human-Model Performance Gap in Text Embedding Taskscited by 7
  12. 2024BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Languagecited by 6
14 more
  1. 2025MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutchcited by 5
  2. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  3. 2025Dynaword: From One-shot to Continuously Developed Datasetscited by 3
  4. 2026Towards Multimodal Time Series Anomaly Detection with Semantic Alignment and Condensed Interactioncited by 3
  5. 2025SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?cited by 2
  6. 2024Benchmarking pre-trained text embedding models in aligning built asset informationcited by 1
  7. 2026MVEB: Massive Video Embedding Benchmarkcited by 1
  8. 2024A Comparative Study of Text Retrieval Models on DaReCzechcited by 0
  9. 2026ML-Embed: Inclusive and Efficient Embeddings for a Multilingual Worldcited by 0
  10. 2026Grounding Text Embeddings in Stakeholder Associationscited by 0
  11. 2026SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptationcited by 0
  12. 2026HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditionscited by 0
  13. 2026MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguesecited by 0
  14. 2026Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languagescited by 0