QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration

arXiv 2505.06481 ↗First posted 10 May 2025cs.LG cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The deployment of mixture-of-experts (MoE) large language models (LLMs) presents significant challenges due to their high memory demands. These challenges become even more pronounced in multi-tenant environments, where shared resources must accommodate multiple models, limiting the effectiveness of conventional virtualization techniques. This paper addresses the problem of efficiently serving multiple fine-tuned MoE-LLMs on a single-GPU. We propose a serving system that employs textitsimilarity-based expert consolidation to reduce the overall memory footprint by sharing similar experts across models. To ensure output quality, we introduce textitruntime partial reconfiguration, dynamically replacing non-expert layers when processing requests from different models. As a result, our approach achieves a competitive output quality while maintaining throughput comparable to serving a single model while incurring a negligible increase in time-to-first-token (TTFT). Experiments on a server with a single NVIDIA A100 GPU (80GB) using Mixtral-8x7B models demonstrate an 85% average reduction in turnaround time compared to NVIDIA's multi-instance GPU (MIG). Furthermore, experiments on Google's Switch Transformer Base-8 model with up to four variants demonstrate the scalability and resilience of our approach in maintaining output quality compared to other model merging baselines, highlighting its effectiveness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 40 of its 53 references that are in the map. First posted on arXiv 10 May 2025.

Builds on

40 of 53 references.

Closest ancestors · 5

Same direction, newest first

Other references · 35

2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2020Measuring Massive Multitask Language Understandingcited by 7,963
2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
2016Pointer Sentinel Mixture Modelscited by 3,096
2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
2024Mixtral of Expertscited by 1,823
2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396

23 more

2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
2021Merging Models with Fisher-Weighted Averagingcited by 621
2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
2019Model Fusion via Optimal Transportcited by 219
2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
2021Scalable and Efficient MoE Training for Multitask Multilingual Modelscited by 76
2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
2024Parameter Competition Balancing for Model Mergingcited by 72
2024Instruction Tuning for Secure Code Generationcited by 68
2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40
2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 23
2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
2024Learning More Generalized Experts by Merging Experts in Mixture-of-Expertscited by 12
2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way