mlsanityData to 14 Oct 2026

Training Sparse Mixture Of Experts Text Embedding Models

arXiv 2502.07972 ↗First posted 11 Feb 2025cs.CL cs.AI cs.IRcited by 32

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Transformer-based text embedding models have improved their performance on benchmarks like MIRACL and BEIR by increasing their parameter counts. However, this scaling approach introduces significant deployment challenges, including increased inference latency and memory usage. These challenges are particularly severe in retrieval-augmented generation (RAG) applications, where large models' increased memory requirements constrain dataset ingestion capacity, and their higher latency directly impacts query-time performance. While causal language models have addressed similar efficiency challenges using Mixture of Experts (MoE) architectures, this approach hasn't been successfully adapted to the general text embedding setting. In this paper, we introduce Nomic Embed v2, the first general purpose MoE text embedding model. Our model outperforms models in the same parameter class on both monolingual and multilingual benchmarks while also maintaining competitive performance with models twice its size. We open-source all code, models, and evaluation data to ensure full reproducibility of our training pipeline at hrefhttps://github.com/nomic-ai/contrastorshttps://github.com/nomic-ai/contrastors.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 28 of its 38 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

28 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free14 Oct
  2. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  3. 2024Contrastive Learning and Mixture of Experts Enables Precise Vector Embeddings28 Jan
  4. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 21

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  12. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
9 more
  1. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  2. 2023Improving Text Embeddings with Large Language Modelscited by 303
  3. 2024Generative Representational Instruction Tuningcited by 233
  4. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  5. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  6. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  7. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 112
  8. 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 84
  9. 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 20.

32 papers.

Sort:

By area:

  1. 2025ReMindRAG: Low-Cost LLM-Guided Knowledge Graph Traversal for Efficient RAGcited by 8
  2. 2025AutoSurvey2: Empowering Researchers with Next Level Automated Literature Surveyscited by 8
  3. 2025Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generationcited by 6
  4. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  5. 2026Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrievalcited by 3
  6. 2026OntoLearner: A Modular Python Library for Ontology Learning with Large Language Modelscited by 3
  7. 2026Embedding World Knowledge into Tabular Models: Towards Best Practices for Embedding Pipeline Designcited by 2
  8. 2025Advancing Arabic Reverse Dictionary Systems: A Transformer-Based Approach with Dataset Construction Guidelinescited by 1
  9. 2025The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasurecited by 1
  10. 2025What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Modelscited by 1
  11. 2026eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agentscited by 1
  12. 2026Zero-Shot Semantic Re-Identification for Autonomous Driving: A VLM Baseline Studycited by 1
20 more
  1. 2026ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairscited by 1
  2. 2026Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budgetcited by 1
  3. 2025Breaking Distortion-free Watermarks in Large Language Modelscited by 0
  4. 2025Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industrycited by 0
  5. 2025ProtoSiTex: Learning Semi-Interpretable Prototypes for Multi-label Text Classificationcited by 0
  6. 2025TurkEmbed: Turkish Embedding Model on NLI & STS Taskscited by 0
  7. 2025Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometrycited by 0
  8. 2026Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Modelscited by 0
  9. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  10. 2026IdioLink: Retrieving Meaning Beyond Words Across Idiomatic and Literal Expressionscited by 0
  11. 2026MIMO: Multilingual Information Retrieval via Monolingual Objectivescited by 0
  12. 2026SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptationcited by 0
  13. 2026Temporal Preference Optimization for Unsupervised Retrievalcited by 0
  14. 2026HistoRAG: Embedding Historical Methodology in Retrieval-Augmented Generation Through Critical Technical Practicecited by 0
  15. 2026A Comparative Study on Affective Cues in Text Embeddings Across Psychological Emotion Theoriescited by 0
  16. 2026MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQAcited by 0
  17. 2026A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Datasetcited by 0
  18. 2026Don't Contrast the Impossible: Region-Constrained Batching for Contrastive User Modeling on a Local Community Platformcited by 0
  19. 2026What Makes a Good Fiqh Retriever? Answer Retrieval for Arabic Islamic Jurisprudencecited by 0
  20. 2026Mixture-of-Experts Language Models Can Be Strong and Efficient Retrieverscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
  2. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark
  3. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systems
  4. 2025EmbeddingGemma: Powerful and Lightweight Text Representations
  5. 2024Extracting Sentence Embeddings from Pretrained Transformer Models
  6. 2025Mixture of Experts Approaches in Dense Retrieval Tasks
  7. 2024A Survey on Mixture of Experts in Large Language Models
  8. 2025KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model