Binary Neural Networks for Large Language Model: A Survey
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have wide applications in the field of natural language processing(NLP), such as GPT-4 and Llama. However, with the exponential growth of model parameter sizes, LLMs bring significant resource overheads. Low-bit quantization, as a key technique, reduces memory usage and computational demands by decreasing the bit-width of model parameters, activations, and gradients. Previous quantization methods for LLMs have largely employed Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ does not require any retraining of the original model, while QAT involves optimizing precision during training to achieve the best quantization parameters. The BitNet team proposed a radically different approach, where quantization is performed from the start of model training, utilizing low-precision binary weights during the training process. This approach has led to the emergence of many binary quantization techniques for large language models. This paper provides a comprehensive review of these binary quantization techniques. Specifically, we will introduce binary quantization techniques in deep neural networks and further explore their application to LLMs, reviewing their various contributions, implementations, and applications.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 67 of its 85 references that are in the map. First posted on arXiv 26 Feb 2025.
Builds on
67 of 85 references.
References · 67
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2016Layer Normalizationcited by 5,579
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
38 more
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2023PaLM 2 Technical Reportcited by 1,154
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
- 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
- 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
- 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
- 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2015Deep Learning with Limited Numerical Precisioncited by 368
- 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
- 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
- 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
- 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
- 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
- 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
- 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
- 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
- 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
- 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
- 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
- 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
- 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
- 2022Gradients without Backpropagationcited by 73
- 2022BiBERT: Accurate Fully Binarized BERTcited by 70
- 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
- 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
- 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
- 2022BiT: Robustly Binarized Multi-distilled Transformercited by 51
- 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
- 2019QKD: Quantization-aware Knowledge Distillationcited by 42
- 2022Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networkscited by 22
- 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 22
- 2020Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networkscited by 20
50 most cited of 67 All 67 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.