mlsanityData to 7 Oct 2026

AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models

arXiv 2406.13233 ↗First posted 19 Jun 2024cs.AIcited by 41

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture of experts (MoE) has become the standard for constructing production-level large language models (LLMs) due to its promise to boost model capacity without causing significant overheads. Nevertheless, existing MoE methods usually enforce a constant top-k routing for all tokens, which is arguably restrictive because various tokens (e.g., "<EOS>" vs. "apple") may require various numbers of experts for feature abstraction. Lifting such a constraint can help make the most of limited resources and unleash the potential of the model for downstream tasks. In this sense, we introduce AdaMoE to realize token-adaptive routing for MoE, where different tokens are permitted to select a various number of experts. AdaMoE makes minimal modifications to the vanilla MoE with top-k routing -- it simply introduces a fixed number of null experts, which do not consume any FLOPs, to the expert set and increases the value of k. AdaMoE does not force each token to occupy a fixed number of null experts but ensures the average usage of the null experts with a load-balancing loss, leading to an adaptive number of null/true experts used by each token. AdaMoE exhibits a strong resemblance to MoEs with expert choice routing while allowing for trivial auto-regressive modeling. AdaMoE is easy to implement and can be effectively applied to pre-trained (MoE-)LLMs. Extensive studies show that AdaMoE can reduce average expert load (FLOPs) while achieving superior performance. For example, on the ARC-C dataset, applying our method to fine-tuning Mixtral-8x7B can reduce FLOPs by 14.5% while increasing accuracy by 1.69%.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 21 of its 39 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

21 of 39 references.

References · 21

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2024Mixtral of Expertscited by 1,785
  11. 2017Decoupled Weight Decay Regularizationcited by 1,722
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
9 more
  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  2. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  3. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  4. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  5. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  6. 2024Higher Layers Need More LoRA Expertscited by 101
  7. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  8. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34
  9. 2019SocialIQA: Commonsense Reasoning about Social Interactionscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 18; 2026: 17.

41 papers.

Sort:

By area:

  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  2. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  3. 2025LongCat-Flash Technical Reportcited by 54
  4. 2025Advancing Expert Specialization for Better MoEcited by 37
  5. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  6. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  7. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 12
  8. 2025From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Modelscited by 11
  9. 2025DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanismcited by 10
  10. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Expertscited by 10
  11. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
  12. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Expertscited by 9
29 more
  1. 2025Kairos: Toward Adaptive and Parameter-Efficient Time Series Foundation Modelscited by 8
  2. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  3. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  4. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inferencecited by 5
  5. 2024LaDiMo: Layer-wise Distillation Inspired MoEfiercited by 4
  6. 2024Glider: Global and Local Instruction-Driven Expert Routercited by 4
  7. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 4
  8. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillationcited by 4
  9. 2026Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancingcited by 3
  10. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 3
  11. 2024DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Modelscited by 2
  12. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuningcited by 2
  13. 2025THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translationcited by 2
  14. 2025GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Modelscited by 2
  15. 2026RANGER: Sparsely-Gated Mixture-of-Experts with Adaptive Retrieval Re-ranking for Pathology Report Generationcited by 2
  16. 2025Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcyclingcited by 1
  17. 2026Improving MoE Compute Efficiency by Composing Weight and Data Sparsitycited by 1
  18. 2026BEAM: Binary Expert Activation Masking for Dynamic Routing in MoEcited by 1
  19. 2026PAMAE: Phase-Aware-MoE Action Experts Towards Reliable Flow-Matching Vision-Language-Action Policiescited by 1
  20. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routingcited by 1
  21. 2026LoGo: Token-Level Dynamic Local-Global Attentioncited by 1
  22. 2026T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning with Dynamic Routingcited by 1
  23. 2025MoE-PHDS: One MoE checkpoint for flexible runtime sparsitycited by 0
  24. 2026Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Modelscited by 0
  25. 2026ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Expertscited by 0
  26. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEscited by 0
  27. 2026Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Modelscited by 0
  28. 2026MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Modelscited by 0
  29. 2026X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depthscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Layerwise Recurrent Router for Mixture-of-Experts
  2. 2025Route Experts by Sequence, not by Token
  3. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMs
  4. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  5. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  6. 2022Mixture-of-Experts with Expert Choice Routing
  7. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference
  8. 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts