mlsanityData to 14 Oct 2026

Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

arXiv 2608.23806 ↗First posted 24 Aug 2026cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

We introduce Giga-Embeddings, a family of text embedding models designed to combine strong retrieval quality with efficient serving. Its largest member is a sparse 10B-parameter Mixture-of-Experts encoder with approximately 1.8B active parameters per token. Across English, Russian, multilingual, and code MTEB benchmarks, this model achieves the strongest aggregate performance within the family on all four evaluated suites. In our vLLM benchmark with 1024-token inputs, it processes 114.5k tokens per second, providing 25 percent higher throughput than the dense 3B model and 1.56-2.65x the throughput of the evaluated external systems. The family also includes a dense 3B encoder and a distilled 480M encoder for tighter compute and memory budgets. We train the compact model using a dimension-agnostic objective that aligns teacher and student similarity distributions. The resulting 480M model scores 70.98 on Russian MTEB, surpassing FRIDA while using 42 percent fewer parameters. We release all three model checkpoints.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 34 of its 36 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

34 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  2. 2024Mixtral of Experts8 Jan
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 31

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  9. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  10. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  11. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  12. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
19 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  2. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  3. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  4. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  5. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  6. 2023Improving Text Embeddings with Large Language Modelscited by 303
  7. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
  8. 2024Generative Representational Instruction Tuningcited by 233
  9. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 176
  10. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 145
  11. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 125
  12. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 94
  13. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 83
  14. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  15. 2026jina-embeddings-v5-text: Task-Targeted Embedding Distillationcited by 22
  16. 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
  17. 2023DistillCSE: Distilled Contrastive Learning for Sentence Embeddingscited by 4
  18. 2025GUIDE: Guided Initialization and Distillation of Embeddingscited by 1
  19. 2025GigaEmbeddings: Efficient Russian Language Embedding Modelcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Training Sparse Mixture Of Experts Text Embedding Models
  2. 2025EmbeddingGemma: Powerful and Lightweight Text Representations
  3. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark
  4. 2024Extracting Sentence Embeddings from Pretrained Transformer Models
  5. 2025Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit
  6. 2026Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems
  7. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup
  8. 2024A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokens