mlsanityData to 7 Oct 2026

Route Experts by Sequence, not by Token

arXiv 2511.06494 ↗First posted 9 Nov 2025cs.LG cs.AI cs.ITcited by 5

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed number of experts to all tokens, ignoring their varying complexity. Prior adaptive routing methods introduce additional modules and hyperparameters, often requiring costly retraining from scratch. We propose Sequence-level TopK (SeqTopK), a minimal modification that shifts the expert budget from the token level to the sequence level. By selecting the top T · K experts across all T tokens, SeqTopK enables end-to-end learned dynamic allocation -- assigning more experts to difficult tokens and fewer to easy ones -- while preserving the same overall budget. SeqTopK requires only a few lines of code, adds less than 1% overhead, and remains fully compatible with pretrained MoE models. Experiments across math, coding, law, and writing show consistent improvements over TopK and prior parameter-free adaptive methods, with gains that become substantially larger under higher sparsity (up to 16.9%). These results highlight SeqTopK as a simple, efficient, and scalable routing strategy, particularly well-suited for the extreme sparsity regimes of next-generation LLMs. Code is available at https://github.com/Y-Research-SBU/SeqTopK.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 60 of its 74 references that are in the map. First posted on arXiv 9 Nov 2025.

Builds on

60 of 74 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing1 May
  2. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing19 Dec
  3. 2024MoH: Multi-Head Attention as Mixture-of-Head Attention15 Oct
  4. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts28 Aug
  5. 2024GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory18 Jun
  6. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models23 May
  7. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Models12 Mar
  8. 2022StableMoE: Stable Routing Strategy for Mixture of Experts18 Apr
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
38 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  10. 2018Universal Transformerscited by 613
  11. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  12. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  13. 2025Kimi K2: Open Agentic Intelligencecited by 523
  14. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  15. 2022Confident Adaptive Language Modelingcited by 297
  16. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  17. 2022Matryoshka Representation Learningcited by 268
  18. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  19. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  20. 2021Hash Layers For Large Sparse Modelscited by 236
  21. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  22. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 157
  23. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  24. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  25. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 136
  26. 2024BatchTopK Sparse Autoencoderscited by 126
  27. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  28. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  29. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  30. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  31. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  32. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  33. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 25
  34. 2025DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformerscited by 23
  35. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  36. 2025Uncovering Memorization Effect in the Presence of Spurious Correlationscited by 20
  37. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Expertscited by 13
  38. 2023Implicit Anatomical Rendering for Medical Image Segmentation with Stochastic Expertscited by 12

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoningcited by 4
  2. 2025Reflection Pretraining Enables Token-Level Self-Correction in Biological Sequence Modelscited by 0
  3. 2026Hierarchical Copula-Gumbel-Top-K Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Lawscited by 0
  4. 2026Towards a Statistical Understanding of Mixture-of-Expertscited by 0
  5. 2026Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMs
  2. 2026Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
  3. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  4. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  5. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  6. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts
  7. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
  8. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts