mlsanityData to 7 Oct 2026

Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection

arXiv 2411.08982 ↗First posted 13 Nov 2024cs.LG cs.DCcited by 16

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Selective parameter activation provided by Mixture-of-Expert (MoE) models have made them a popular choice in modern foundational models. However, MoEs face a fundamental tension when employed for serving. Batching, critical for performance in serving, forces the activation of all experts, thereby negating MoEs' benefits and exacerbating memory bandwidth bottlenecks. Existing work on efficient MoE inference are unable to resolve this tension even with extensive workload-specific tuning. We present LYNX, a system that enables efficient MoE inference in a workload-agnostic fashion. LYNX leverages a key property of MoE training: load-balancing losses introduce batch-level expert activation skews and redundancy, which it exploits by remapping low-affinity token-to-expert assignments within each batch using a novel AffinityBinning technique that reduces the total experts invoked. Our evaluation of LYNX on four state-of-the-art model families across nine benchmarks shows that it achieves up to 1.30x improvement in throughput while maintaining accuracy loss of less than 1% points across tasks. Further, LYNX is complementary to existing techniques where it additionally boosts their performance by up to 1.38x.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 31 of its 63 references that are in the map. First posted on arXiv 13 Nov 2024.

Builds on

31 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  2. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  3. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  4. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  5. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun

Other references · 26

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
14 more
  1. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  2. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  3. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  4. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  5. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  6. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  7. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  8. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  9. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
  10. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  11. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 36
  12. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  13. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  14. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 8.

16 papers.

Sort:

By area:

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  2. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  3. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  4. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  5. 2026XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inferencecited by 7
  6. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  7. 2025Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retrainingcited by 6
  8. 2026SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Modelscited by 3
  9. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computingcited by 2
  10. 2026ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Servingcited by 2
  11. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Accelerationcited by 2
  12. 2025Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inferencecited by 1
4 more
  1. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  2. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inferencecited by 0
  3. 2026SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Servingcited by 0
  4. 2026BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  2. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference
  3. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference
  4. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
  5. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
  6. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  7. 2026PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition
  8. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts