mlsanityData to 14 Oct 2026

QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models

arXiv 2310.16795 ↗First posted 25 Oct 2023cs.LGcited by 42

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures offer a general solution to the high inference costs of large language models (LLMs) via sparse routing, bringing faster and more accurate models, at the cost of massive parameter counts. For example, the SwitchTransformer-c2048 model has 1.6 trillion parameters, requiring 3.2TB of accelerator memory to run efficiently, which makes practical deployment challenging and expensive. In this paper, we present a solution to this memory problem, in form of a new compression and execution framework called QMoE. Specifically, QMoE consists of a scalable algorithm which accurately compresses trillion-parameter MoEs to less than 1 bit per parameter, in a custom format co-designed with bespoke GPU decoding kernels to facilitate efficient end-to-end compressed inference, with minor runtime overheads relative to uncompressed execution. Concretely, QMoE can compress the 1.6 trillion parameter SwitchTransformer-c2048 model to less than 160GB (20x compression, 0.8 bits per parameter) at only minor accuracy loss, in less than a day on a single GPU. This enables, for the first time, the execution of a trillion-parameter model on affordable commodity hardware, like a single server with 4x NVIDIA A6000 or 8x NVIDIA 3090 GPUs, at less than 5% runtime overhead relative to ideal uncompressed inference. The source code and compressed models are available at github.com/IST-DASLab/qmoe.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 38 of its 50 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

38 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness3 Oct
  2. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs16 Aug
  3. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production18 Nov
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  6. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 31

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  12. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
19 more
  1. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  2. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  3. 2021A White Paper on Neural Network Quantizationcited by 472
  4. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  5. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  6. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  7. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  8. 2023From Sparse to Soft Mixtures of Expertscited by 248
  9. 2021Hash Layers For Large Sparse Modelscited by 237
  10. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  11. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  12. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
  13. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  14. 2019Fast Sparse ConvNetscited by 86
  15. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 82
  16. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  17. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  18. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  19. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 16; 2026: 14.

42 papers.

Sort:

By area:

  1. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  2. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 91
  3. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  4. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  5. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  6. 2024zkLLM: Zero Knowledge Proofs for Large Language Modelscited by 51
  7. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  8. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  9. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 36
  10. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMscited by 22
  11. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 19
  12. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
30 more
  1. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decodingcited by 12
  2. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  3. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 10
  4. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  5. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Servicecited by 7
  6. 2024Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Modelscited by 7
  7. 2025Exploiting Edited Large Language Models as General Scientific Optimizerscited by 7
  8. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inferencecited by 7
  9. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMscited by 7
  10. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantizationcited by 7
  11. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 6
  12. 2025From Low Intrinsic Dimensionality to Non-Vacuous Generalization Bounds in Deep Multi-Task Learningcited by 5
  13. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  14. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  15. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  16. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  17. 2025EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devicescited by 2
  18. 2024One-Bit Quantization and Sparsification for Multiclass Linear Classification with Strong Regularizationcited by 1
  19. 2025Physics-Inspired Binary Neural Networks: Interpretable Compression with Theoretical Guaranteescited by 1
  20. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 1
  21. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  22. 2026TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tilingcited by 1
  23. 2026Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deploymentcited by 1
  24. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
  25. 2025Tiny-QMoEcited by 0
  26. 2026ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbitscited by 0
  27. 2026Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Servingcited by 0
  28. 2026Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMscited by 0
  29. 2026Infinite-Parameter LLMs: Generating and Adapting Weights from Live Datacited by 0
  30. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Stagingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
  2. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
  3. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
  4. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
  5. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration
  6. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
  7. 2026MonoMoE: An Efficient Fused Mega-kernel for Quantized MoE Decoding
  8. 2025Collaborative Compression for Large-Scale MoE Deployment on Edge