# Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection

arXiv: https://arxiv.org/abs/2411.08982 · HTML: https://mlsanity.com/paper/2411.08982/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 31 of its 63 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 30 of 31)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [Tutel: Adaptive Mixture-of-Experts at Scale](https://mlsanity.com/paper/2206.03382/) (2022)
- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://mlsanity.com/paper/2408.00741/) (2024)
- [Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy](https://mlsanity.com/paper/2310.01334/) (2023)
- [Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference](https://mlsanity.com/paper/2308.12066/) (2023)
- [Fast Inference of Mixture-of-Experts Language Models with Offloading](https://mlsanity.com/paper/2312.17238/) (2023)
- [Task-Specific Expert Pruning for Sparse Mixture-of-Experts](https://mlsanity.com/paper/2206.00277/) (2022)
- [Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models](https://mlsanity.com/paper/2402.07033/) (2024)
- [MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router](https://mlsanity.com/paper/2410.12013/) (2024)
- [MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition](https://mlsanity.com/paper/2411.01016/) (2024)
- [Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models](https://mlsanity.com/paper/2405.14297/) (2024)
- [Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs](https://mlsanity.com/paper/2407.00945/) (2024)
- [SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts](https://mlsanity.com/paper/2404.05089/) (2024)
- [SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention](https://mlsanity.com/paper/2406.15486/) (2024)
- [Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference](https://mlsanity.com/paper/2303.06182/) (2023)
- [STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning](https://mlsanity.com/paper/2409.06211/) (2024)
- [Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks](https://mlsanity.com/paper/2405.15731/) (2024)

## Built on by (most cited first, 16 of 16)

- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading](https://mlsanity.com/paper/2502.05370/) (2025)
- [Taming the Titans: A Survey of Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.19720/) (2025)
- [LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers](https://mlsanity.com/paper/2509.23638/) (2025)
- [Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining](https://mlsanity.com/paper/2511.02237/) (2025)
- [XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference](https://mlsanity.com/paper/2602.07265/) (2026)
- [Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference](https://mlsanity.com/paper/2510.05497/) (2025)
- [SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models](https://mlsanity.com/paper/2602.07616/) (2026)
- [ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving](https://mlsanity.com/paper/2607.00466/) (2026)
- [Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing](https://mlsanity.com/paper/2512.18674/) (2025)
- [ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration](https://mlsanity.com/paper/2608.24938/) (2026)
- [Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inference](https://mlsanity.com/paper/2508.09229/) (2025)
- [Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling](https://mlsanity.com/paper/2603.27624/) (2026)
- [Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference](https://mlsanity.com/paper/2607.05116/) (2026)
- [SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving](https://mlsanity.com/paper/2609.34117/) (2026)
- [BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding](https://mlsanity.com/paper/2609.36222/) (2026)

## Abstract (arXiv)

Selective parameter activation provided by Mixture-of-Expert (MoE) models have made them a popular choice in modern foundational models. However, MoEs face a fundamental tension when employed for serving. Batching, critical for performance in serving, forces the activation of all experts, thereby negating MoEs' benefits and exacerbating memory bandwidth bottlenecks. Existing work on efficient MoE inference are unable to resolve this tension even with extensive workload-specific tuning. We present LYNX, a system that enables efficient MoE inference in a workload-agnostic fashion. LYNX leverages a key property of MoE training: load-balancing losses introduce batch-level expert activation skews and redundancy, which it exploits by remapping low-affinity token-to-expert assignments within each batch using a novel AffinityBinning technique that reduces the total experts invoked. Our evaluation of LYNX on four state-of-the-art model families across nine benchmarks shows that it achieves up to 1.30x improvement in throughput while maintaining accuracy loss of less than 1% points across tasks. Further, LYNX is complementary to existing techniques where it additionally boosts their performance by up to 1.38x.