# Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs

arXiv: https://arxiv.org/abs/2604.07888 · HTML: https://mlsanity.com/paper/2604.07888/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Builds on (most cited first, 27 of 27)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [SqueezeLLM: Dense-and-Sparse Quantization](https://mlsanity.com/paper/2306.07629/) (2023)
- [Improving Neural Network Quantization without Retraining using Outlier Channel Splitting](https://mlsanity.com/paper/1901.09504/) (2019)
- [BiLLM: Pushing the Limit of Post-Training Quantization for LLMs](https://mlsanity.com/paper/2402.04291/) (2024)
- [BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation](https://mlsanity.com/paper/2402.10631/) (2024)
- [PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression](https://mlsanity.com/paper/2405.14852/) (2024)
- [Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs](https://mlsanity.com/paper/2402.10517/) (2024)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [BitNet b1.58 2B4T Technical Report](https://mlsanity.com/paper/2504.12285/) (2025)
- [Delta Decompression for MoE-based LLMs Compression](https://mlsanity.com/paper/2502.17298/) (2025)
- [Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging](https://mlsanity.com/paper/2506.23266/) (2025)
- [Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization](https://mlsanity.com/paper/2404.03605/) (2024)
- [QuEST: Stable Training of LLMs with 1-Bit Weights and Activations](https://mlsanity.com/paper/2502.05003/) (2025)
- [Spectra: Surprising Effectiveness of Pretraining Ternary Language Models at Scale](https://mlsanity.com/paper/2407.12327/) (2024)
- [BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs](https://mlsanity.com/paper/2504.18415/) (2025)
- [FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation](https://mlsanity.com/paper/2407.07093/) (2024)
- [Rotate, Clip, and Partition: Towards W2A4KV4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizer](https://mlsanity.com/paper/2502.15779/) (2025)
- [BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook](https://mlsanity.com/paper/2506.12040/) (2025)
- [CTMQ: Cyclic Training of Convolutional Neural Networks with Multiple Quantization Steps](https://mlsanity.com/paper/2206.12794/) (2022)

## Built on by (most cited first, 3 of 3)

- [IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference](https://mlsanity.com/paper/2605.25475/) (2026)
- [OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit](https://mlsanity.com/paper/2609.31631/) (2026)
- [Linger and Lose: Knowledge Collapse in Low-Bit Language Models](https://mlsanity.com/paper/2609.32902/) (2026)

## Abstract (arXiv)

Training LLMs at ultra-low precision remains a formidable challenge. Direct low-bit QAT often suffers from convergence instability and substantial training costs, exacerbated by quantization noise from heavy-tailed outlier channels and error accumulation across layers. To address these issues, we present Bit-by-Bit, a progressive QAT framework with outlier channel splitting. Our approach integrates three key components: (1) block-wise progressive training that reduces precision stage by stage, ensuring stable initialization for low-bit optimization; (2) nested structure of integer quantization grids to enable a "train once, deploy any precision" paradigm, allowing a single model to support multiple bit-widths without retraining; (3) rounding-aware outlier channel splitting, which mitigates quantization error while acting as an identity transform that preserves the quantized outputs. Furthermore, we follow microscaling groups with E4M3 scales, capturing dynamic activation ranges in alignment with OCP/NVIDIA standards. To address the lack of efficient 2-bit kernels, we developed custom operators for both W2A2 and W2A16 configurations, achieving up to 11× speedup over BF16. Under W2A2 settings, Bit-by-Bit significantly outperforms baselines like BitDistiller and EfficientQAT on both Llama2/3, achieving a loss of only 2.25 WikiText2 PPL compared to full-precision models.