mlsanityData to 7 Oct 2026

CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval

arXiv 2311.02834 ↗First posted 6 Nov 2023cs.IRcited by 6

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The first-stage retrieval aims to retrieve a subset of candidate documents from a huge collection both effectively and efficiently. Since various matching patterns can exist between queries and relevant documents, previous work tries to combine multiple retrieval models to find as many relevant results as possible. The constructed ensembles, whether learned independently or jointly, do not care which component model is more suitable to an instance during training. Thus, they cannot fully exploit the capabilities of different types of retrieval models in identifying diverse relevance patterns. Motivated by this observation, in this paper, we propose a Mixture-of-Experts (MoE) model consisting of representative matching experts and a novel competitive learning mechanism to let the experts develop and enhance their expertise during training. Specifically, our MoE model shares the bottom layers to learn common semantic representations and uses differently structured upper layers to represent various types of retrieval experts. Our competitive learning mechanism has two stages: (1) a standardized learning stage to train the experts equally to develop their capabilities to conduct relevance matching; (2) a specialized learning stage where the experts compete with each other on every training instance and get rewards and updates according to their performance to enhance their expertise on certain types of samples. Experimental results on three retrieval benchmark datasets show that our method significantly outperforms the state-of-the-art baselines.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 42 of its 73 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

42 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  7. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  9. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  11. 2020Overview of the TREC 2019 deep learning trackcited by 282
  12. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
29 more
  1. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  2. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 230
  3. 2019Document Expansion by Query Predictioncited by 225
  4. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  5. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  6. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  7. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  8. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  9. 2017A Deep Relevance Matching Model for Ad-hoc Retrievalcited by 126
  10. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  11. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  12. 2021Adversarial Retriever-Ranker for dense text retrievalcited by 82
  13. 2021One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Predictioncited by 78
  14. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 59
  15. 2021Semantic Models for the First-stage Retrieval: A Comprehensive Reviewcited by 57
  16. 2020RepBERT: Contextualized Text Embeddings for First-Stage Retrievalcited by 46
  17. 2020SparTerm: Learning Term-based Sparse Representation for Fast Text Retrievalcited by 45
  18. 2020Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approachcited by 31
  19. 2021TREC Deep Learning Track: Reusable Test Collections in the Large Data Regimecited by 23
  20. 2022UnifieR: A Unified Retriever for Large-Scale Retrievalcited by 19
  21. 2021Shallow pooling for sparse labelscited by 13
  22. 2022Automatic Expert Selection for Multi-Scenario and Multi-Task Searchcited by 9
  23. 2022MASTER: Multi-task Pre-trained Bottlenecked Masked Autoencoders are Better Dense Retrieverscited by 7
  24. 2020Adversarial Mixture Of Experts with Category Hierarchy Soft Constraintcited by 6
  25. 2021Densifying Sparse Representations for Passage Retrieval by Representational Slicingcited by 6
  26. 2022Mixture of Experts for Biomedical Question Answeringcited by 6
  27. 2022Hard Negatives or False Negatives: Correcting Pooling Bias in Training Neural Ranking Modelscited by 6
  28. 2021Query Embedding Pruning for Dense Retrievalcited by 5
  29. 2023CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrievalcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2024Multi-granular Adversarial Attacks against Black-box Neural Ranking Modelscited by 14
  2. 2024Investigating Mixture of Experts in Dense Retrievalcited by 4
  3. 2026Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrievalcited by 3
  4. 2025Mixture of Experts Approaches in Dense Retrieval Taskscited by 2
  5. 2026SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translationcited by 1
  6. 2025Are Information Retrieval Approaches Good at Harmonising Longitudinal Survey Questions in Social Science?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers
  2. 2025ExpertRAG: Efficient RAG with Mixture of Experts -- Optimizing Context Retrieval for Adaptive LLM Responses
  3. 2025Orchestrating Heterogeneous Experts: A Scalable MoE Framework with Anisotropy-Preserving Fusion
  4. 2025Mixture-of-Retrieval Experts for Reasoning-Guided Multimodal Knowledge Exploitation
  5. 2021ExpertRank: A Multi-level Coarse-grained Expert-based Listwise Ranking Loss
  6. 2024Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs
  7. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
  8. 2026Training Dense Retrievers with Multiple Positive Passages