mlsanityData to 14 Oct 2026

Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models

arXiv 2508.06974 ↗First posted 9 Aug 2025cs.CLcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

1-bit LLM quantization offers significant advantages in reducing storage and computational costs. However, existing methods typically train 1-bit LLMs from scratch, failing to fully leverage pre-trained models. This results in high training costs and notable accuracy degradation. We identify that the large gap between full precision and 1-bit representations makes naive adaptation difficult. In this paper, we introduce a consistent progressive training for both forward and backward, smoothly converting the full-precision weights into the binarized ones. Additionally, we incorporate binary-aware initialization and dual-scaling compensation to reduce the difficulty of progressive training and improve the performance. Experimental results on LLMs of various sizes demonstrate that our method outperforms existing approaches. Our results show that high-performance 1-bit LLMs can be achieved using pre-trained models, eliminating the need for expensive training from scratch.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 44 of its 48 references that are in the map. First posted on arXiv 9 Aug 2025.

Builds on

44 of 48 references.

References · 44

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2023A Survey of Large Language Modelscited by 2,576
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
32 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  6. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  7. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  8. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  9. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  10. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  11. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  12. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  13. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  14. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  15. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  16. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  17. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 104
  18. 2023PB-LLM: Partially Binarized Large Language Modelscited by 83
  19. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  20. 2020Rotated Binary Neural Networkcited by 77
  21. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55
  22. 2024A Survey on Transformer Compressioncited by 46
  23. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  24. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  25. 2025BitNet b1.58 2B4T Technical Reportcited by 38
  26. 2024DB-LLM: Accurate Dual-Binarization for Efficient LLMscited by 37
  27. 2023CBQ: Cross-Block Quantization for Large Language Modelscited by 34
  28. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Modelscited by 31
  29. 2025An Empirical Study of Qwen3 Quantizationcited by 21
  30. 2024Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokenscited by 18
  31. 2024Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Modelscited by 14
  32. 2024FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weightscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs
  2. 2025SDQ-LLM: Sigma-Delta Quantization for 1-bit LLMs of any size
  3. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization
  4. 2025Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?
  5. 2025An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits
  6. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorization
  7. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs
  8. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?