mlsanityData to 7 Oct 2026

Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems

arXiv 2605.23618 ↗First posted 22 May 2026cs.CLcited by 0

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

We benchmark Google Embeddings (GE2), a Vertex-AI-hosted bi-encoder with 2,048-token context and explicit task-type conditioning, against five open-source alternatives: BGE-M3, E5-large, Multilingual-E5-large (mE5-L), LaBSE, and Paraphrase-Multilingual-MPNet (mMPNet). Evaluation covers four BEIR subsets, a synthetic Italian RAG corpus, a chunking ablation considering 5 sizes of tokens with three strategies, and per-query latency on commodity CPU hardware. GE2 ranks first on every task, achieving BEIR avg.nDCG@10 = 0.638 and IT-RAG-Bench nDCG@10 = 0.282, but at 231.6 ms median latency, it is roughly 14x slower than the fastest local models. mE5-L reaches within 0.003 nDCG of GE2 on Italian at 31 ms, making it the preferred option when sub-100 ms SLAs matter. A more striking finding concerns LaBSE, which, despite widespread multilingual deployment scores 0.188 average nDCG@10 on BEIR, below every dedicated retrieval model including mMPNet. Chunking experiments show that all six models saturate at 32-token chunks on our corpus, with semantic chunking providing measurable gains only at 16 tokens.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 10 of its 12 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

10 of 12 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Context Embeddings for Efficient Answer Generation in RAG12 Jul
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 8

  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  2. 2017Billion-scale similarity search with GPUscited by 2,477
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  5. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  6. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  7. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  8. 2020Language-agnostic BERT Sentence Embeddingcited by 621

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
  2. 2026Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models
  3. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark
  4. 2025MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
  5. 2025Training Sparse Mixture Of Experts Text Embedding Models
  6. 2025KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model
  7. 2026Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
  8. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems