mlsanityData to 7 Oct 2026

MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models

arXiv 2609.08663 ↗First posted 8 Sep 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities with increased complexity. Prior scaling methods either increase the representation size, retrieval effort, or scales the encoder into a heavy multimodal LLM. Recent works, such as Think-Then-Embed (TTE), explore scaling via reasoning tokens. However, embedding models are hard to scale up: increasing parameters directly tradeoffs for the large training batch size that contrastive learning needs, and retrieval has to be served under tight latency. Moreover, UME tasks are diverse in complexity, where scaling up embedders can bring significant redundant computation. In this work, we propose MOEMB, which instead scales UME along the expert axis through mixture-of-experts (MoE), growing encoder capacity while preserving single-vector, non-autoregressive encoding. Through a systematic study of the design space and training recipes for MoE-based UME, MoEMB sets a new state of the art on both MMEB-V2 and MRMR among models trained on public MMEB-family data: with only 3B active parameters, MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes. To further improve the scalability and efficiency, we conduct the first comprehensive study of adaptive computation for MoE-based embedding, spanning diverse strategies across training-based and inference-only methods. Together, these results support expert scaling as an effective and efficient direction for UME, with adaptive computation further improving efficiency for MLLM-based embedding models towards large-scale retrieval and recommendation systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 55 of its 59 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

55 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings5 Mar
  2. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping19 Nov
  3. 2024Aria: An Open Multimodal Native Mixture-of-Experts Model8 Oct

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  10. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  11. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  12. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
38 more
  1. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  4. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  6. 2022Matryoshka Representation Learningcited by 268
  7. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  8. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  9. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  10. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  11. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  12. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  13. 2021PonderNet: Learning to Pondercited by 156
  14. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  15. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  16. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  17. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  18. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  19. 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 41
  20. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 41
  21. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 40
  22. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 39
  23. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  24. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  25. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
  26. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 27
  27. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  28. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
  29. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 22
  30. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  31. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  32. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 14
  33. 2025MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrievalcited by 12
  34. 2026PLUME: Latent Reasoning Based Universal Multimodal Embeddingcited by 10
  35. 20242D Matryoshka Sentence Embeddingscited by 8
  36. 2025Reason to Contrast: A Cascaded Multimodal Retrieval Frameworkcited by 7
  37. 2026MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Modelscited by 6
  38. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  2. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
  3. 2026Scaling Embeddings Outperforms Scaling Experts in Language Models
  4. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
  5. 2025Compressing then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding
  6. 2026Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
  7. 2026LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models
  8. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs