mlsanityData to 7 Oct 2026

Mixture of A Million Experts

arXiv 2407.04153 ↗First posted 4 Jul 2024cs.LG cs.AIcited by 70

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. Sparse mixture-of-experts (MoE) architectures have emerged as a viable approach to address this issue by decoupling model size from computational cost. The recent discovery of the fine-grained MoE scaling law shows that higher granularity leads to better performance. However, existing MoE models are limited to a small number of experts due to computational and optimization challenges. This paper introduces PEER (parameter efficient expert retrieval), a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of tiny experts (over a million). Experiments on language modeling tasks demonstrate that PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off. By enabling efficient utilization of a massive number of experts, PEER unlocks the potential for further scaling of transformer models while maintaining computational efficiency.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 36 of its 41 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

36 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Fast Feedforward Networks28 Aug
  2. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 33

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
21 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  4. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  5. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  6. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  7. 2021Hash Layers For Large Sparse Modelscited by 236
  8. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  9. 2017The Case for Learned Index Structurescited by 188
  10. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  11. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  12. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  13. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  14. 2019Large Memory Layers with Product Keyscited by 120
  15. 2013Deep Learning of Representations: Looking Forwardcited by 84
  16. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  17. 2023Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generationcited by 54
  18. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  19. 2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44
  20. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  21. 2023Memory Augmented Language Models through Mixture of Word Expertscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 27; 2026: 26.

70 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  4. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  5. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  6. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Modelscited by 48
  7. 2024Memory Layers at Scalecited by 39
  8. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  9. 2025Mixture of Experts Made Intrinsically Interpretablecited by 22
  10. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  11. 2024Ultra-Sparse Memory Networkcited by 19
  12. 2025Scaling Embedding Layers in Language Modelscited by 19
38 more
  1. 2025Continual Learning via Sparse Memory Finetuningcited by 17
  2. 2026Intelligent AI Delegationcited by 16
  3. 2024Monet: Mixture of Monosemantic Experts for Transformerscited by 12
  4. 2024TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competitioncited by 10
  5. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  6. 2026STEM: Scaling Transformers with Embedding Modulescited by 10
  7. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 9
  8. 2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7
  9. 2025Mixture of Neuron Expertscited by 7
  10. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Levelcited by 7
  11. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  12. 2024A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentationcited by 6
  13. 2025Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Expertscited by 5
  14. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  15. 2025Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoderscited by 5
  16. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  17. 2025UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learningcited by 5
  18. 2025On the Origin of Algorithmic Progress in AIcited by 5
  19. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  20. 2024Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Modelscited by 4
  21. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  22. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  23. 2025Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsitycited by 3
  24. 2026MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformerscited by 3
  25. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Expertscited by 3
  26. 2024Interacting Large Language Model Agents. Interpretable Models and Social Learningcited by 2
  27. 2024Mixture of Hidden-Dimensions Transformercited by 2
  28. 2025Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Taskscited by 2
  29. 2026JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulationcited by 2
  30. 2026Modularity is the Bedrock of Natural and Artificial Intelligencecited by 2
  31. 2026Temporally Extended Mixture-of-Experts Modelscited by 2
  32. 2024Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecturecited by 1
  33. 2024BIG-MoE: Bypass Isolated Gating MoE for Generalized Multimodal Face Anti-Spoofingcited by 1
  34. 2025ExplicitLM: Decoupling Knowledge from Parameters via Explicit Memory Bankscited by 1
  35. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Qualitycited by 1
  36. 2026UniPool: A Globally Shared Expert Pool for Mixture-of-Expertscited by 1
  37. 2024Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Taskscited by 0
  38. 2024MoIN: Mixture of Introvert Experts to Upcycle an LLMcited by 0

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework
  2. 2025Secret mixtures of experts inside your LLM
  3. 2026DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models
  5. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
  6. 2026Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers
  7. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  8. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection