mlsanityData to 14 Oct 2026

BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs

arXiv 2507.17133 ↗First posted 23 Jul 2025cs.DCcited by 4

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

In recent years, the Mixture-of-Experts (MoE) architecture has been widely applied to large language models (LLMs), providing a promising solution that activates only a subset of the model's parameters during computation, thereby reducing overall memory requirements and allowing for faster inference compared to dense models. Despite these advantages, existing systems still face issues of low efficiency due to static model placement and lack of dynamic workloads adaptation. This leads to suboptimal resource utilization and increased latency, especially during bursty requests periods. To address these challenges, this paper introduces BrownoutServe, a novel serving framework designed to optimize inference efficiency and maintain service reliability for MoE-based LLMs under dynamic computational demands and traffic conditions. BrownoutServe introduces "united experts" that integrate knowledge from multiple experts, reducing the times of expert access and inference latency. Additionally, it proposes a dynamic brownout mechanism to adaptively adjust the processing of certain tokens, optimizing inference performance while guaranteeing service level objectives (SLOs) are met. Our evaluations show the effectiveness of BrownoutServe under various workloads: it achieves up to 2.07x throughput improvement compared to vLLM and reduces SLO violations by 90.28%, showcasing its robustness under bursty traffic while maintaining acceptable inference accuracy.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 27 of its 50 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

27 of 50 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  3. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  4. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  5. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 21

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2024Mixtral of Expertscited by 1,823
9 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  8. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  9. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025EC2MoE: Adaptive End-Cloud Pipeline Collaboration Enabling Scalable Mixture-of-Experts Inferencecited by 3
  2. 2026SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharingcited by 2
  3. 2026SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PCcited by 0
  4. 2026DScale: Scaling Block-Diffusion Speculative Decoding with Adaptive Verificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BrownoutMoE: Structure-Aware Expert Grouping for Efficient and Accurate LLM Web-based Services
  2. 2025Faster MoE LLM Inference for Extremely Large Models
  3. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
  4. 2026MoEless: Efficient MoE LLM Serving via Serverless Computing
  5. 2025ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models
  6. 2026Multi-Layer Scheduling for MoE-Based LLM Reasoning
  7. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
  8. 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference