mlsanityData to 14 Oct 2026

Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs

arXiv 2604.07888 ↗First posted 9 Apr 2026cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Training LLMs at ultra-low precision remains a formidable challenge. Direct low-bit QAT often suffers from convergence instability and substantial training costs, exacerbated by quantization noise from heavy-tailed outlier channels and error accumulation across layers. To address these issues, we present Bit-by-Bit, a progressive QAT framework with outlier channel splitting. Our approach integrates three key components: (1) block-wise progressive training that reduces precision stage by stage, ensuring stable initialization for low-bit optimization; (2) nested structure of integer quantization grids to enable a "train once, deploy any precision" paradigm, allowing a single model to support multiple bit-widths without retraining; (3) rounding-aware outlier channel splitting, which mitigates quantization error while acting as an identity transform that preserves the quantized outputs. Furthermore, we follow microscaling groups with E4M3 scales, capturing dynamic activation ranges in alignment with OCP/NVIDIA standards. To address the lack of efficient 2-bit kernels, we developed custom operators for both W2A2 and W2A16 configurations, achieving up to 11× speedup over BF16. Under W2A2 settings, Bit-by-Bit significantly outperforms baselines like BitDistiller and EfficientQAT on both Llama2/3, achieving a loss of only 2.25 WikiText2 PPL compared to full-precision models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

27 of 40 references.

References · 27

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  7. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  8. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  9. 2025Muon is Scalable for LLM Trainingcited by 428
  10. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  11. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 151
  12. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
15 more
  1. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  2. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  3. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 54
  4. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  5. 2025BitNet b1.58 2B4T Technical Reportcited by 38
  6. 2025Delta Decompression for MoE-based LLMs Compressioncited by 33
  7. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 28
  8. 2024Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularizationcited by 23
  9. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 22
  10. 2024Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scalecited by 21
  11. 2025BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMscited by 18
  12. 2024FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillationcited by 12
  13. 2025Rotate, Clip, and Partition: Towards W2A4KV4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizercited by 4
  14. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebookcited by 4
  15. 2022CTMQ: Cyclic Training of Convolutional Neural Networks with Multiple Quantization Stepscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inferencecited by 7
  2. 2026OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuitcited by 0
  3. 2026Linger and Lose: Knowledge Collapse in Low-Bit Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20261-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization
  2. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMs
  3. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantization
  4. 2026CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
  5. 2026A Target-Centric Survey of Quantization-Aware Training
  6. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
  7. 2024Accurate Block Quantization in LLMs with Outliers
  8. 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference