mlsanityData to 14 Oct 2026

MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Design

arXiv 2412.14590 ↗First posted 19 Dec 2024cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Quantization has become one of the most effective methodologies to compress LLMs into smaller size. However, the existing quantization solutions still show limitations of either non-negligible accuracy drop or low system efficiency. In this paper, we propose MixLLM that explores the optimization space of mixed-precision quantization between output features, based on the insight that different features matter differently in the model. MixLLM identifies the important output features in the global view rather than within each single layer, effectively assigning larger bit-width to output features that need it the most to achieve high accuracy and low memory usage. We present the sweet spot of quantization configuration of algorithm-system co-design with high accuracy and system efficiency. To address the system challenge, we design the two-step dequantization to make use of the Tensor Core easily and fast data type conversion to reduce dequantization overhead, and present the software pipeline to overlap the memory access, dequantization and the MatMul to the best. Extensive experiments show that with only 10% more bits, the perplexity increase can be reduced from about 0.5 in SOTA to within 0.2 for Llama 3.1 70B, while MMLU-Pro loss can be reduced from 1.92 to 0.99 over the SOTA of three popular models. Besides its superior accuracy, MixLLM also achieves state-of-the-art system efficiency. Code is released at https://github.com/microsoft/MixLLM.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

41 of 50 references.

References · 41

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  12. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
29 more
  1. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  3. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  5. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  6. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  7. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  8. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  9. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  10. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  11. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  12. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  13. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  14. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  15. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201
  16. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  17. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  18. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  19. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  20. 2024Scaling Laws for Precisioncited by 95
  21. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  22. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  23. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 63
  24. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  25. 2018A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Datasetcited by 55
  26. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
  27. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17
  28. 2023ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Taskscited by 14
  29. 2022Demystifying the Nvidia Ampere Architecture through Microbenchmarking and Instruction-level Analysiscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 10.

16 papers.

Sort:

By area:

  1. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
  2. 2026MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelscited by 5
  3. 2025You Had One Job: Per-Task Quantization Using LLMs' Hidden Representationscited by 3
  4. 2026Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deploymentcited by 3
  5. 2026SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Trainingcited by 2
  6. 2025Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Methodcited by 1
  7. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Modelscited by 1
  8. 2026Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Modelscited by 1
  9. 2025Bridging Technology and Humanities: Evaluating the Impact of Large Language Models on Social Sciences Research with DeepSeek-R1cited by 0
  10. 2025SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMscited by 0
  11. 2025SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMscited by 0
  12. 2026RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inferencecited by 0
4 more
  1. 2026RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learningcited by 0
  2. 2026FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inferencecited by 0
  3. 2026A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraintcited by 0
  4. 2026FoldQuantVLA: Native Low-Bit Quantization of Vision-Language-Action Models via Consistent Foldingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
  2. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs
  3. 2025FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs
  4. 2026Beyond Outliers: A Data-Free Layer-wise Mixed-Precision Quantization Approach Driven by Numerical and Structural Dual-Sensitivity
  5. 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
  6. 2023A Speed Odyssey for Deployable Quantization of LLMs
  7. 2024Channel-Wise Mixed-Precision Quantization for Large Language Models
  8. 2026APreQEL: Adaptive Mixed Precision Quantization For Edge LLMs