Quality-Constrained Routing over a Fixed Pool of Quantized Mixture-of-Experts Instances

arXiv 2609.12550 ↗First posted 11 Sep 2026cs.LGcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Quantized Mixture-of-Experts (MoE) services can hold several pre-materialized instances of one base model, but quantization damage varies sharply across requests and bitwidths. Because instance materialization and replica counts consume memory and require slow reconfiguration, we treat them as upstream provisioning decisions and study routing within a fixed resident pool. Within this fixed-pool boundary, we route each request to maximize modeled throughput under a class-level expected quality-degradation budget and measured instance capacities. To predict this request-specific risk, we introduce FWP (Fragility-Weighted Perplexity), computed from prompt tokens on a reference-instance prefill and calibrated to candidate-instance degradation. Underlying FWP is an exact two-expert affinity--fragility decomposition and a conditional multi-layer top-k expansion whose bias, interaction, route-change, separability, and higher-order terms remain explicit. Using these calibrated risks, a window-level linear program yields a signed reduced-reward score that is KKT-consistent with the LP optimum under optimal prices and primal-feasible tie allocation. On 88 extended Qwen prompts, complete W2, W3, and W4 instances quantizing all 6,144 expert blocks incur mean DeltaNLL of 0.9437, 0.1832, and 0.0513. Under the same population and τ=0.1513, FWP allocation reaches a 1.284× offline model-based multiplier versus 1.253× for request-agnostic mixing and 1.000× for static W4, an incremental 2.5% relative FWP gain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 49 of its 61 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

49 of 61 references.

Closest ancestors · 2

Same direction, newest first

Other references · 47

2025Qwen3 Technical Reportcited by 8,820
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2017On Calibration of Modern Neural Networkscited by 4,960
2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
2024Qwen2 Technical Reportcited by 2,120
2024Mixtral of Expertscited by 1,823
2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141

35 more

2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
2017Selective Classification for Deep Neural Networkscited by 852
2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
2024SpinQuant: LLM quantization with learned rotationscited by 471
2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
2022Confident Adaptive Language Modelingcited by 310
2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
2021Hash Layers For Large Sparse Modelscited by 237
2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
2023Conformal Language Modelingcited by 170
2024Language Models with Conformal Factuality Guaranteescited by 131
2023AutoMix: Automatically Mixing Language Modelscited by 125
2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 41
2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guaranteescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way