mlsanityData to 14 Oct 2026

Statistically-Lossless Quantization of Large Language Models

arXiv 2605.02404 ↗First posted 4 May 2026cs.LGcited by 6

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Model quantization has become essential for efficient large language model deployment, yet existing approaches present clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but lack inference acceleration. This paper explores the middle ground of statistically-lossless compression, examining three complementary aspects of what losslessness means for quantized LLMs. First, task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and is achievable at aggressive bitwidths. Second, we formalize the stricter notion of distribution-lossless compression, requiring the quantized model's next-token distribution to be practically indistinguishable from the original, and propose the Expected Acceptance Rate (EAR), the maximum token-agreement probability under optimal coupling, as a directly interpretable fidelity metric. For example, EAR >= 0.99 means 99% agreement. Third, we prove a gamma-squared variance law showing that symmetric quantization inflates noise variance by gamma^2 relative to asymmetric quantization, making asymmetric quantization a prerequisite for distribution-lossless fidelity but not for task-level preservation. Through SLQ, a layer-wise non-uniform method with asymmetric quantization and wide bitwidth search, we obtain task-lossless compression at well below 4 bits per parameter, as low as 3.3 bits depending on the model, distribution-lossless compression at 5-6 bits per parameter on average, and inference speedups of 1.7-3.7x compared to FP16 using optimized kernels. Source code is available at [https://github.com/IST-DASLab/SLQ](https://github.com/IST-DASLab/SLQ).

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 49 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

35 of 49 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024SpinQuant: LLM quantization with learned rotations26 May
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar

Other references · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  9. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  10. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  11. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  12. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
21 more
  1. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  2. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  3. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  4. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  5. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  6. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  7. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 52
  8. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  9. 2024Investigating Layer Importance in Large Language Modelscited by 36
  10. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  11. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 24
  12. 2025Model-Preserving Adaptive Roundingcited by 23
  13. 2024EvoPress: Accurate Dynamic Model Compression via Evolutionary Searchcited by 20
  14. 2024ZipNN: Lossless Compression for AI Modelscited by 14
  15. 2025Efficient Shapley Value-based Non-Uniform Pruning of Large Language Modelscited by 8
  16. 2026When LLMs get significantly worse: A statistical approach to detect model degradationscited by 7
  17. 2023QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Modelscited by 6
  18. 2025To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentrationcited by 5
  19. 2026The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost Morecited by 4
  20. 2026Model Compression with Exact Budget Constraints via Riemannian Manifoldscited by 3
  21. 2024Shapley Pruning for Neural Network Compressioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Grid Games: The Power of Multiple Grids for Quantizing Large Language Modelscited by 2
  2. 2026Certifying Compressed Language Models: An Audit and a Statistical Toolkitcited by 2
  3. 2026Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Servingcited by 0
  4. 2026A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Qualitycited by 0
  5. 2026G2PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensationcited by 0
  6. 2026Q-PACE: Dynamic Precision Allocation for Quantization-Aware Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression
  2. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theorem
  3. 2026Sliced-Wasserstein Distribution Alignment Loss Improves the Ultra-Low-Bit Quantization of Large Language Models
  4. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Size
  5. 2024When Quantization Affects Confidence of Large Language Models?
  6. 2025When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models
  7. 2023A Survey on Model Compression for Large Language Models
  8. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Models