mlsanityData to 7 Oct 2026

SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs

arXiv 2512.05409 ↗First posted 5 Dec 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs). However, existing low-bit quantization and sparsification techniques are difficult to balance accuracy and efficiency due to the limited hardware support. For example, W4A8 can only achieve the same peak TOPS as W8A8 whereas the GPU-supported sparse data format (2:4 semi-structure sparse) is seldomly adopted due to the loss of accuracy. To bridge this gap, in this paper, we propose the Sparse-Quantized Format (SQ-format), which is a unified data format for quantization and sparsification potentially easily supported by new hardware and existing GPUs. SQ-format makes use of the fact that sparse matrix can be accelerated in high-precision, and low-precision matrix multiplication can also be accelerated accordingly. As such, SQ-format is proposed to achieve Pareto improvement between performance and throughput. This format is particularly suitable for activations with outlier inequality status and makes their static compression possible. We show the state-of-the-art PTQ performance with SQ-format, propose the hardware required to support it, and further offer the design exploration and insights for the next-generation AI accelerators.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 5 Dec 2025.

Builds on

25 of 36 references.

References · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  12. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
13 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  2. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  3. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  5. 2022FP8 Formats for Deep Learningcited by 187
  6. 2023Microscaling Data Formats for Deep Learningcited by 132
  7. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  8. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  9. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  10. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 20
  11. 2021S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Accelerationcited by 19
  12. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  13. 2024Ascend HiFloat8 Format for Deep Learningcited by 7

Built on by

Not cited yet.