mlsanityData to 14 Oct 2026

Routing in Sparsely-gated Language Models responds to Context

arXiv 2409.14107 ↗First posted 21 Sep 2024cs.CLcited by 1

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Language Models (LMs) recently incorporate mixture-of-experts layers consisting of a router and a collection of experts to scale up their parameter count given a fixed computational budget. Building on previous efforts indicating that token-expert assignments are predominantly influenced by token identities and positions, we trace routing decisions of similarity-annotated text pairs to evaluate the context sensitivity of learned token-expert assignments. We observe that routing in encoder layers mainly depends on (semantic) associations, but contextual cues provide an additional layer of refinement. Conversely, routing in decoder layers is more variable and markedly less sensitive to context.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 31 of its 42 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

31 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts12 Dec
  2. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  3. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  4. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate29 Dec
  5. 2021Taming Sparsely Activated Transformer with Stochastic Experts8 Oct
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2024Mixtral of Expertscited by 1,823
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  12. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
13 more
  1. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  2. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  3. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  5. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  6. 2023From Sparse to Soft Mixtures of Expertscited by 248
  7. 2021Hash Layers For Large Sparse Modelscited by 237
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  9. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  10. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  11. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22
  12. 2022Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformerscited by 22
  13. 2020Detecting Multiword Expression Type Helps Lexical Complexity Assessmentcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Multi-level context Modeling for consistent expert selection in Mixture-of-Expertscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Probing Semantic Routing in Large Mixture-of-Expert Models
  2. 2026Geometric Routing Enables Causal Expert Control in Mixture of Experts
  3. 2026A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models
  4. 2025Multilingual Routing in Mixture-of-Experts
  5. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
  6. 2026Mixture of Experts for Low-Resource LLMs
  7. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
  8. 2026Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts