mlsanityData to 14 Oct 2026

Post Training Quantization of Large Language Models with Microscaling Formats

arXiv 2405.07135 ↗First posted 12 May 2024cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have distinguished themselves with outstanding performance in complex language modeling tasks, yet they come with significant computational and storage challenges. This paper explores the potential of quantization to mitigate these challenges. We systematically study the combined application of three well-known post-training techniques, SmoothQuant, AWQ, and GPTQ, and provide a comprehensive analysis of their interactions and implications for advancing LLM quantization. We enhance the versatility of these methods by enabling quantization to microscaling (MX) formats, extending the applicability of these PTQ algorithms beyond their original fixed-point format targets. We show that combining different PTQ methods enables us to quantize models to 4-bit weights and 8-bit activations using the MXINT format with negligible accuracy loss compared to the uncompressed baseline.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 12 May 2024.

Builds on

31 of 48 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
19 more
  1. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  4. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  6. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  8. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  9. 2021A White Paper on Neural Network Quantizationcited by 472
  10. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  11. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  12. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  13. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  14. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  15. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  16. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  17. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  18. 2023Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantizationcited by 20
  19. 2024Accurate Block Quantization in LLMs with Outlierscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 8.

12 papers.

  1. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 36
  2. 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Modelscited by 11
  3. 2026M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantizationcited by 8
  4. 2025You Had One Job: Per-Task Quantization Using LLMs' Hidden Representationscited by 3
  5. 2026LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMscited by 2
  6. 2026QMC: Efficient SLM Edge Inference via Outlier-Aware Quantization and Emergent Memories Co-Designcited by 1
  7. 2026MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inferencecited by 1
  8. 2026Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Modelscited by 1
  9. 2025LLM Compression: How Far Can We Go in Balancing Size and Performance?cited by 0
  10. 2026HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inferencecited by 0
  11. 2026Predicting Quantization Price for Selecting PTQ Configurations Before Deploymentcited by 0
  12. 2026From Grey-Box to Green-Box: When can Physics-Informed Machine Learning Reduce Carbon Footprints in Structural Health Monitoring?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
  2. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Models
  3. 2025Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models
  4. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
  5. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
  6. 2025PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models
  7. 2024Evaluating Quantized Large Language Models
  8. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices