mlsanityData to 7 Oct 2026

SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs

arXiv 2512.04746 ↗First posted 4 Dec 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Extremely low-bit quantization is critical for efficiently deploying Large Language Models (LLMs), yet it often leads to severe performance degradation at 2 bits and even at 4 bits (e.g., MXFP4). We present SignRoundV2, a post-training quantization framework designed to maintain high performance even under aggressive compression. SignRoundV2 introduces (1) a simple yet efficient adaptive mixed-precision strategy that leverages gradient information and quantization-induced reconstruction errors to guide layer-wise bit allocation, and (2) a set of lightweight stabilization techniques, including loss filtering and a pre-tuning scale search, to improve tuning effectiveness in extremely low-bit regimes. Our approach takes a significant step toward closing the performance gap between quantized and full-precision models. Experimental results across diverse LLMs demonstrate that SignRoundV2 achieves near-lossless performance in mixed MXFP settings, narrowing the gap to ~1% at an average of 4.5 bits, while substantially improving accuracy in challenging 2-bit weight-only quantization. The source code is available at urlhttps://github.com/intel/auto-round.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 55 of its 73 references that are in the map. First posted on arXiv 4 Dec 2025.

Builds on

55 of 73 references.

References · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  8. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  10. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  12. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  13. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  14. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  15. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  16. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  17. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  18. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  19. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  20. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  21. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  22. 2021Post-Training Quantization for Vision Transformercited by 274
  23. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  24. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  25. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  26. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 89
  27. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  28. 2017Adaptive Quantization for Deep Neural Networkcited by 67
  29. 2019AutoQ: Automated Kernel-Wise Neural Network Quantizationcited by 60
  30. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 55
  31. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 49
  32. 2023FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantizationcited by 46
  33. 2024Scaling Laws for Forgetting When Fine-Tuning Large Language Modelscited by 36
  34. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 27
  35. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  36. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  37. 2025AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Modelscited by 15
  38. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.