mlsanityData to 14 Oct 2026

BitNet: Scaling 1-bit Transformers for Large Language Models

arXiv 2310.11453 ↗First posted 17 Oct 2023cs.CLcited by 189

Not on the map. Why?

Abstract (arXiv)

The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. Furthermore, BitNet exhibits a scaling law akin to full-precision Transformers, suggesting its potential for effective scaling to even larger language models while maintaining efficiency and performance benefits.

Key facts. Cited by 189 papers in the map (data through 14 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

29 of 35 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2016Layer Normalizationcited by 5,669
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
17 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  4. 2018Group Normalizationcited by 1,524
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2023PaLM 2 Technical Reportcited by 1,162
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  9. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  10. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  11. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  12. 2019XNOR-Net++: Improved Binary Neural Networkscited by 97
  13. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  14. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 49
  15. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  16. 2022Foundation Transformerscited by 19
  17. 2023Binarized Neural Machine Translationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 53; 2025: 79; 2026: 56.

189 papers.

Sort:

By area:

  1. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  2. 2023Efficient Large Language Models: A Surveycited by 186
  3. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  4. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  5. 2024Scaling Laws for Precisioncited by 95
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  7. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  8. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  9. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 60
  10. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  11. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  12. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55
38 more
  1. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  2. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
  3. 2024Training of Physical Neural Networkscited by 49
  4. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  5. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  6. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 43
  7. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 41
  8. 2024TelecomGPT: A Framework to Build Telecom-Specfic Large Language Modelscited by 40
  9. 2025FP4 All the Way: Fully Quantized Training of LLMscited by 40
  10. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 39
  11. 2025BitNet b1.58 2B4T Technical Reportcited by 38
  12. 2024SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanismscited by 31
  13. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  14. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  15. 20241.58-bit FLUXcited by 31
  16. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  17. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Surveycited by 26
  18. 2024Scalable MatMul-free Language Modelingcited by 25
  19. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 25
  20. 2024Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustnesscited by 22
  21. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 22
  22. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 21
  23. 2024Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokenscited by 18
  24. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 18
  25. 2025BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMscited by 18
  26. 2024BitNet a4.8: 4-bit Activations for 1-bit LLMscited by 16
  27. 2024Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Modelscited by 14
  28. 2024GenAINet: Enabling Wireless Collective Intelligence via Knowledge Transfer and Reasoningcited by 13
  29. 2025When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Modelscited by 13
  30. 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
  31. 2024FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillationcited by 12
  32. 20241-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUscited by 12
  33. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  34. 2025Compression Scaling Laws:Unifying Sparsity and Quantizationcited by 11
  35. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  36. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  37. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  38. 2025TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAscited by 10

50 most cited of 189 All 189 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization
  2. 2024Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniques
  3. 2025Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?
  4. 2025An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits
  5. 2025Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models
  6. 2026Variable-Width Transformers
  7. 2025Large Language Model Partitioning for Low-Latency Inference at the Edge
  8. 2026pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training