mlsanityData to 14 Oct 2026

MTEB-French: Resources for French Sentence Embedding Evaluation and Analysis

arXiv 2405.20468 ↗First posted 30 May 2024cs.CL cs.IR cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Recently, numerous embedding models have been made available and widely used for various NLP tasks. The Massive Text Embedding Benchmark (MTEB) has primarily simplified the process of choosing a model that performs well for several tasks in English, but extensions to other languages remain challenging. This is why we expand MTEB to propose the first massive benchmark of sentence embeddings for French. We gather 15 existing datasets in an easy-to-use interface and create three new French datasets for a global evaluation of 8 task categories. We compare 51 carefully selected embedding models on a large scale, conduct comprehensive statistical tests, and analyze the correlation between model performance and many of their characteristics. We find out that even if no model is the best on all tasks, large multilingual models pre-trained on sentence similarity perform exceptionally well. Our work comes with open-source code, new datasets and a public leaderboard.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 30 of its 49 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

30 of 49 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  10. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
18 more
  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  2. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  3. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  4. 2019CamemBERT: a Tasty French Language Modelcited by 368
  5. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
  6. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  7. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  8. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  9. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
  10. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  11. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  12. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  13. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 53
  14. 2021A Statutory Article Retrieval Dataset in Frenchcited by 40
  15. 2020A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Modelscited by 30
  16. 2018Open Subtitles Paraphrase Corpus for Six Languagescited by 24
  17. 2024German Text Embedding Clustering Benchmarkcited by 13
  18. 2019DiaBLa: A Corpus of Bilingual Spontaneous Written Dialogues for Machine Translationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 3.

6 papers.

  1. 2024Making Text Embedders Few-Shot Learnerscited by 79
  2. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  3. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  4. 2026ML-Embed: Inclusive and Efficient Embeddings for a Multilingual Worldcited by 0
  5. 2026HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditionscited by 0
  6. 2026MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguesecited by 0