mlsanityData to 14 Oct 2026

BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE

arXiv 2605.14438 ↗First posted 14 May 2026cs.AIcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures enhance the efficiency of large language models by activating only a subset of experts per token. However, standard MoE employs a fixed Top-K routing strategy, leading to redundant computation and suboptimal inference latency. Existing acceleration methods either require costly retraining with architectural changes or suffer from severe performance drop at high sparsity due to train-inference mismatch. To address these limitations, we propose BEAM (Binary Expert Activation Masking), a novel method that learns token-adaptive expert selection via trainable binary masks. With a straight-through estimator and an auxiliary regularization loss, BEAM induces dynamic expert sparsity through end-to-end training while maintaining model capability. We further implement an efficient custom CUDA kernel for BEAM, ensuring seamless integration with the vLLM inference framework. Experiments show that BEAM retains over 98% of the original model's performance while reducing MoE layer FLOPs by up to 85%, achieving up to 2.5× faster decoding and 1.4× higher throughput, demonstrating its effectiveness as a practical, plug-and-play solution for efficient MoE inference.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 35 of its 36 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

35 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts9 Oct

Other references · 34

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  11. 2024Mixtral of Expertscited by 1,823
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
22 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  4. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  5. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  6. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  7. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 72
  8. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  9. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 56
  10. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 53
  11. 2024AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Modelscited by 44
  12. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 37
  13. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clusteringcited by 35
  14. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selectioncited by 28
  15. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 22
  16. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  17. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Modelscited by 15
  18. 2025Learning to Skip the Middle Layers of Transformerscited by 10
  19. 2024MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Expertscited by 9
  20. 2025DASH: Input-Aware Dynamic Layer Skipping for Efficient LLM Inference with Markov Decision Policiescited by 3
  21. 2025DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-trainingcited by 3
  22. 2026ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decodingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  2. 2024Expert-Token Resonance MoE: Bidirectional Routing with Efficiency Affinity-Driven Active Selection
  3. 2026Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
  4. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  5. 2026BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding
  6. 2025DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
  7. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
  8. 2026MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling