mlsanityData to 7 Oct 2026

The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design

arXiv 2408.12503 ↗First posted 22 Aug 2024cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Embedding models play a crucial role in Natural Language Processing (NLP) by creating text embeddings used in various tasks such as information retrieval and assessing semantic text similarity. This paper focuses on research related to embedding models in the Russian language. It introduces a new Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark, the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our benchmark includes seven categories of tasks, such as semantic textual similarity, text classification, reranking, and retrieval.The research also assesses a representative set of Russian and multilingual models on the proposed benchmark. The findings indicate that the new model achieves results that are on par with state-of-the-art models in Russian. We release the model ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code, integration into the original framework and a public leaderboard.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 44 of its 62 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

44 of 62 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
32 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  4. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  9. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  10. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  11. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  12. 2021Generating Datasets with Pretrained Language Modelscited by 192
  13. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  14. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
  15. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  16. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  17. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  18. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  19. 2019Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmaxcited by 58
  20. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  21. 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47
  22. 2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
  23. 2019SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysiscited by 38
  24. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  25. 2022RuCoLA: Russian Corpus of Linguistic Acceptabilitycited by 26
  26. 2024The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embeddingcited by 26
  27. 2019Self-Attentive Model for Headline Generationcited by 16
  28. 2023DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Trainingcited by 15
  29. 2021Detecting Inappropriate Messages on Sensitive Topics that Could Harm a Company's Reputationcited by 6
  30. 2023A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languagescited by 5
  31. 2020Automatically Ranked Russian Paraphrase Corpus for Text Generationcited by 2
  32. 2022WikiOmnia: generative QA corpus on the whole Russian Wikipediacited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 7.

20 papers.

Sort:

By area:

  1. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  2. 2025HUME: Measuring the Human-Model Performance Gap in Text Embedding Taskscited by 7
  3. 2024BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Languagecited by 6
  4. 2025DRAGOn: Designing RAG On Periodically Updated Corpuscited by 5
  5. 2025MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutchcited by 5
  6. 2025M3Retrieve: Benchmarking Multimodal Retrieval for Medicinecited by 5
  7. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  8. 2025AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languagescited by 3
  9. 2024Benchmarking and Building Zero-Shot Hindi Retrieval Model with Hindi-BEIR and NLLB-E5cited by 2
  10. 2025Building Russian Benchmark for Evaluation of Information Retrieval Modelscited by 2
  11. 2026VIBE: Visual Instruction Based Editorcited by 2
  12. 2025GigaEmbeddings: Efficient Russian Language Embedding Modelcited by 1
8 more
  1. 2025Large Language Models in the Task of Automatic Validation of Text Classifier Predictionscited by 0
  2. 2025Wikipedia-based Datasets in Russian Information Retrieval Benchmark RusBEIRcited by 0
  3. 2026AI Appeals Processor: A Deep Learning Approach to Automated Classification of Citizen Appeals in Government Servicescited by 0
  4. 2026ML-Embed: Inclusive and Efficient Embeddings for a Multilingual Worldcited by 0
  5. 2026SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptationcited by 0
  6. 2026HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditionscited by 0
  7. 2026MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguesecited by 0
  8. 2026Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddingscited by 0