mlsanityData to 14 Oct 2026

GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling

arXiv 2604.18556 ↗First posted 20 Apr 2026cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of the art is currently split into simple scalar quantization techniques, such as GPTQ or AWQ, which are widely deployed but plateau in accuracy at 3-4 bits per parameter (bpp), and "second-generation" vector- or trellis-quantized methods, such as QTIP, GPTVQ and AQLM, which push the accuracy frontier but are notoriously hard to implement and to scale. In this paper, we ask whether this gap is fundamental, or whether a carefully optimized textitscalar quantizer can recover most of it. We answer in the affirmative, by introducing GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales using a Gumbel-Softmax relaxation of the discrete grid. GSQ matches the cardinality of the relaxation to the small number of levels available in the target bit-width regime (e.g., 3-8 levels for ternary and 3 bpp, respectively), making optimization tractable. Practically, on the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits, while using a symmetric scalar grid with group-wise quantization, and thus remains compatible with existing scalar inference kernels. We further show that the same discrete-assignment optimization can be applied to practical GGUF K-Quant checkpoints: starting from publicly released GGUF models, GSQ improves accuracy while projecting the result back into the same deployment format. Finally, GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply. The source code is publicly available at https://github.com/IST-DASLab/GSQ.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 53 of its 67 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

53 of 67 references.

References · 53

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
38 more
  1. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  2. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  5. 2019Learned Step Size Quantizationcited by 628
  6. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  7. 2025Kimi K2: Open Agentic Intelligencecited by 567
  8. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  9. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  10. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  11. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  12. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  13. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  14. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  15. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  16. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  17. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  18. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  19. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  20. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  21. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  22. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 88
  23. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  24. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  25. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 62
  26. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  27. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 50
  28. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  29. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  30. 2024DB-LLM: Accurate Dual-Binarization for Efficient LLMscited by 37
  31. 2025MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidancecited by 34
  32. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  33. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 31
  34. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18
  35. 2024TernaryLLM: Ternarized Large Language Modelcited by 17
  36. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 10
  37. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Modelscited by 9
  38. 2025Tequila: Trapping-free Ternary Quantization for Large Language Modelscited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Attention Sinks and Outliers in Attention Residualscited by 2
  2. 2026Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recoverycited by 1
  3. 2026Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Modelscited by 1
  4. 2026Quantization Degradation in Large Language Models: A Signal-Noise Perspectivecited by 0
  5. 2026Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Expertscited by 0
  6. 2026Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm
  2. 2024CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization
  3. 2026KronQ: LLM Quantization via Kronecker-Factored Hessian
  4. 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
  5. 2026Scale Sensitivity in Low-Bit Post-Training Quantization: Curvature of the Quantization Error Landscape
  6. 2024EXAQ: Exponent Aware Quantization For LLMs Acceleration
  7. 2026MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models
  8. 2026Two-Stage Grid Optimization for Group-wise Quantization of LLMs