# BitNet b1.58 Reloaded: State-of-the-art Performance Also on Smaller Networks

arXiv: https://arxiv.org/abs/2407.09527 · HTML: https://mlsanity.com/paper/2407.09527/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 12 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023)
- [The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits](https://mlsanity.com/paper/2402.17764/) (2024)
- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2207.01405/) (2022)

## Built on by (most cited first, 8 of 8)

- [Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey](https://mlsanity.com/paper/2508.13073/) (2025)
- [Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification](https://mlsanity.com/paper/2601.07892/) (2026)
- [Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?](https://mlsanity.com/paper/2502.11895/) (2025)
- [Mapping the Schedule x Bit-Width Boundary in Sub-100M Quantisation-Aware Training](https://mlsanity.com/paper/2605.25966/) (2026)
- [When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization](https://mlsanity.com/paper/2411.05882/) (2024)
- [Binary Neural Networks for Large Language Model: A Survey](https://mlsanity.com/paper/2502.19008/) (2025)
- [A Target-Centric Survey of Quantization-Aware Training](https://mlsanity.com/paper/2608.29667/) (2026)
- [Linger and Lose: Knowledge Collapse in Low-Bit Language Models](https://mlsanity.com/paper/2609.32902/) (2026)

## Abstract (arXiv)

Recently proposed methods for 1-bit and 1.58-bit quantization aware training investigate the performance and behavior of these methods in the context of large language models, finding state-of-the-art performance for models with more than 3B parameters. In this work, we investigate 1.58-bit quantization for small language and vision models ranging from 100K to 48M parameters. We introduce a variant of BitNet b1.58, which allows to rely on the median rather than the mean in the quantization process. Through extensive experiments we investigate the performance of 1.58-bit models obtained through quantization aware training. We further investigate the robustness of 1.58-bit quantization-aware training to changes in the learning rate and regularization through weight decay, finding different patterns for small language and vision models than previously reported for large language models. Our results showcase that 1.58-bit quantization-aware training provides state-of-the-art performance for small language models when doubling hidden layer sizes and reaches or even surpasses state-of-the-art performance for small vision models of identical size. Ultimately, we demonstrate that 1.58-bit quantization-aware training is a viable and promising approach also for training smaller deep learning networks, facilitating deployment of such models in low-resource use-cases and encouraging future research.