mlsanityData to 14 Oct 2026

When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization

arXiv 2411.05882 ↗First posted 8 Nov 2024cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Contemporary machine learning models, such as language models, are powerful, but come with immense resource requirements both at training and inference time. It has been shown that decoder-only language models can be trained to a competitive state with ternary weights (1.58 bits per weight), facilitating efficient inference. Here, we start our exploration with non-transformer model architectures, investigating 1.58-bit training for multi-layer perceptrons and graph neural networks. Then, we explore 1.58-bit training in other transformer-based language models, namely encoder-only and encoder-decoder models. Our results show that in all of these settings, 1.58-bit training is on par with or sometimes even better than the standard 32/16-bit models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

20 of 26 references.

References · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2016Layer Normalizationcited by 5,669
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2019Simplifying Graph Convolutional Networkscited by 1,477
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  11. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  12. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
8 more
  1. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  2. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  3. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  4. 2021Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLPcited by 17
  5. 2024BitNet b1.58 Reloaded: State-of-the-art Performance Also on Smaller Networkscited by 8
  6. 2023nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resourcescited by 7
  7. 2024Exploring Extreme Quantization in Spiking Language Modelscited by 4
  8. 2024LLaVaOLMoBitnet1B: Ternary LLM goes Multimodal!cited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?cited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2023BitNet: Scaling 1-bit Transformers for Large Language Models
  2. 2025An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits
  3. 2025Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
  4. 2024BitNet a4.8: 4-bit Activations for 1-bit LLMs
  5. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
  6. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization
  7. 2025PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models
  8. 2026Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models