mlsanityData to 14 Oct 2026

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free

arXiv 2410.10814 ↗First posted 14 Oct 2024cs.CL cs.LGcited by 37

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

While large language models (LLMs) excel on generation tasks, their decoder-only architecture often limits their potential as embedding models if no further representation finetuning is applied. Does this contradict their claim of generalists? To answer the question, we take a closer look at Mixture-of-Experts (MoE) LLMs. Our study shows that the expert routers in MoE LLMs can serve as an off-the-shelf embedding model with promising performance on a diverse class of embedding-focused tasks, without requiring any finetuning. Moreover, our extensive analysis shows that the MoE routing weights (RW) is complementary to the hidden state (HS) of LLMs, a widely-used embedding. Compared to HS, we find that RW is more robust to the choice of prompts and focuses on high-level semantics. Motivated by the analysis, we propose MoEE combining RW and HS, which achieves better performance than using either separately. Our exploration of their combination and prompting strategy shed several novel insights, e.g., a weighted sum of RW and HS similarities outperforms the similarity on their concatenation. Our experiments are conducted on 6 embedding tasks with 20 datasets from the Massive Text Embedding Benchmark (MTEB). The results demonstrate the significant improvement brought by MoEE to LLM-based embedding without further finetuning.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 35 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

27 of 35 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  3. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models24 May
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 23

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  7. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  8. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  9. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  10. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  11. 2015Skip-Thought Vectorscited by 322
  12. 2023Improving Text Embeddings with Large Language Modelscited by 303
11 more
  1. 2024Generative Representational Instruction Tuningcited by 233
  2. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  3. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  4. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  5. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  6. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  7. 2023Scaling Sentence Embeddings with Large Language Modelscited by 85
  8. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  9. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  10. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 62
  11. 2024Meta-Task Prompting Elicits Embeddings from Large Language Modelscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 20; 2026: 14.

37 papers.

Sort:

By area:

  1. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  2. 2025Training Sparse Mixture Of Experts Text Embedding Modelscited by 32
  3. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 26
  4. 2025R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expertscited by 12
  5. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMscited by 12
  6. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  7. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  8. 2025GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecturecited by 8
  9. 2025FreeRet: MLLMs as Training-Free Retrieverscited by 8
  10. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  11. 2025GRACE: Generative Representation Learning via Contrastive Policy Optimizationcited by 7
  12. 2025Interpretable Depression Detection from Social Media Text Using LLM-Derived Embeddingscited by 6
25 more
  1. 2025Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systemscited by 6
  2. 2026The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Modelscited by 6
  3. 2025Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activationscited by 5
  4. 2025Medical Reasoning in LLMs: An In-Depth Analysis of DeepSeek R1cited by 5
  5. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  6. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  7. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compressioncited by 3
  8. 2025Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddingscited by 3
  9. 2026LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden Statescited by 3
  10. 2025THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translationcited by 2
  11. 2025Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRIcited by 2
  12. 2025AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cachecited by 2
  13. 2026Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEscited by 2
  14. 2026MIPIC: Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chainingcited by 2
  15. 2025Rethinking Sparse Mixture of Experts from a Unified Perspectivecited by 1
  16. 2025MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Securitycited by 1
  17. 2026Eigenvectors of Experts are Training-free Non-collapsing Routerscited by 1
  18. 2026Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddingscited by 1
  19. 2026Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learningcited by 1
  20. 2026CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapterscited by 0
  21. 2026Lightweight Chunk Selection for Mobile Retrieval-Augmented Generationcited by 0
  22. 2026TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptationcited by 0
  23. 2026MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Modelscited by 0
  24. 2026Mixture-of-Experts Language Models Can Be Strong and Efficient Retrieverscited by 0
  25. 2026Cross-Lingual Alignment for Decoder-Only Models using MoE Routerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  2. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  3. 2025Advancing Expert Specialization for Better MoE
  4. 2025Mixture of Experts in Large Language Models
  5. 2026MoRE: Mixture of Reused Experts
  6. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
  7. 2024A Survey on Mixture of Experts in Large Language Models
  8. 2025Exploiting the Experts: Unauthorized Compression in MoE-LLMs