mlsanityData to 14 Oct 2026

SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability

arXiv 2610.14745 ↗First posted 10 Oct 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quantization often suffers severe accuracy loss, while vector quantization (VQ) better preserves accuracy through more expressive joint encoding. However, existing VQ methods still exhibit substantial degradation on real-world tasks, and two design challenges remain underexplored: (1) protecting sensitive weight channels within jointly encoded vectors; and (2) allocating a fixed bit budget among representation components, such as direction and radius, to suit different weight matrices. To address these challenges, we propose SAGE: Split Adaptation and Grouping by Energy for weight-only vector quantization. First, Energy-Stratified Grouping (ESG) couples channel grouping with activation-weighted fitting, grouping channels from different energy tiers within each subvector to prioritize sensitive channels. Second, Adaptive Bit Split (ABS) balances direction-codebook capacity and radius precision per matrix, selecting the split with the lowest activation-weighted reconstruction error under a fixed budget. Across benchmarks covering knowledge, mathematics, code generation, instruction following, and tool use, SAGE achieves state-of-the-art performance in the 2-bit quantization setting. On Qwen3-30B-A3B, it retains 95.3% of the BF16 model's average benchmark score. With our custom-designed CUDA kernel, SAGE delivers up to a 2.55-fold non-streaming end-to-end speedup over FP16. The code is available at https://github.com/Zincwolf/sage.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 52 of its 58 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

52 of 58 references.

References · 52

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
38 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  3. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  4. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  5. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  6. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  7. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  8. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  9. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  10. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
  11. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  13. 2024Are We Done with MMLU?cited by 194
  14. 2026Kimi K3: Open Frontier Intelligencecited by 138
  15. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  16. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  17. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 97
  18. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  19. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  20. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 82
  21. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  22. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 60
  23. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 60
  24. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  25. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  26. 2025When2Call: When (not) to Call Toolscited by 42
  27. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 41
  28. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 31
  29. 2025NestQuant: Nested Lattice Quantization for Matrix Products and LLMscited by 23
  30. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 22
  31. 2025ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inferencecited by 17
  32. 2025PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decouplingcited by 7
  33. 2026Leech Lattice Vector Quantization for Efficient LLM Compressioncited by 7
  34. 2026Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantizationcited by 5
  35. 2025PT2-LLM: Post-Training Ternarization for Large Language Modelscited by 4
  36. 2026LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantizationcited by 3
  37. 2026UniSVQ: 2-bit Unified Scalar-Vector Quantizationcited by 2
  38. 2026TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tilingcited by 1

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.