mlsanityData to 7 Oct 2026

DESIRE-ME: Domain-Enhanced Supervised Information REtrieval using Mixture-of-Experts

arXiv 2403.13468 ↗First posted 20 Mar 2024cs.IRcited by 7

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Open-domain question answering requires retrieval systems able to cope with the diverse and varied nature of questions, providing accurate answers across a broad spectrum of query types and topics. To deal with such topic heterogeneity through a unique model, we propose DESIRE-ME, a neural information retrieval model that leverages the Mixture-of-Experts framework to combine multiple specialized neural models. We rely on Wikipedia data to train an effective neural gating mechanism that classifies the incoming query and that weighs the predictions of the different domain-specific experts correspondingly. This allows DESIRE-ME to specialize adaptively in multiple domains. Through extensive experiments on publicly available datasets, we show that our proposal can effectively generalize domain-enhanced neural models. DESIRE-ME excels in handling open-domain questions adaptively, boosting by up to 12% in NDCG@10 and 22% in P@1, the underlying state-of-the-art dense retrieval model.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 18 of its 31 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

18 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 17

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2016Language Modeling with Gated Convolutional Networkscited by 688
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  10. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  11. 2023From Sparse to Soft Mixtures of Expertscited by 245
  12. 2019Document Expansion by Query Predictioncited by 225
5 more
  1. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  2. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  3. 2022No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrievalcited by 17
  4. 2023Dense Retrieval Adaptation using Target Domain Descriptioncited by 8
  5. 2022Mixture of Experts for Biomedical Question Answeringcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspectivecited by 15
  2. 2024Investigating Mixture of Experts in Dense Retrievalcited by 4
  3. 2025Investigating Task Arithmetic for Zero-Shot Information Retrievalcited by 4
  4. 2025When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIRcited by 3
  5. 2025Mixture of Experts Approaches in Dense Retrieval Taskscited by 2
  6. 2026Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learningcited by 0
  7. 2026Mixture-of-Experts Language Models Can Be Strong and Efficient Retrieverscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answering
  2. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering
  3. 2022Variational Open-Domain Question Answering
  4. 2025Orchestrating Heterogeneous Experts: A Scalable MoE Framework with Anisotropy-Preserving Fusion
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
  6. 2024RouterRetriever: Routing over a Mixture of Expert Embedding Models
  7. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
  8. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models