mlsanityData to 7 Oct 2026

ARB-LLM: Alternating Refined Binarizations for Large Language Models

arXiv 2410.03129 ↗First posted 4 Oct 2024cs.CV cs.AI cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLMX and ARB-LLMRC respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLMRC is the first to surpass FP16 models of the same size. The code and models will be available at https://github.com/ZHITENGLI/ARB-LLM.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 35 of its 40 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

35 of 40 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
23 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  6. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  8. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  9. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  10. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  11. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  12. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  13. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  14. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 137
  15. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  16. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  17. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  18. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  19. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 77
  20. 2024Revisiting Knowledge Distillation for Autoregressive Language Modelscited by 41
  21. 2018Alternating Multi-bit Quantization for Recurrent Neural Networkscited by 30
  22. 2017Network Sketching: Exploiting Binary Structure in Deep CNNscited by 22
  23. 2024Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 11.

30 papers.

Sort:

By area:

  1. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 44
  2. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 21
  3. 2025DVD-Quant: Data-free Video Diffusion Transformers Quantizationcited by 14
  4. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  5. 2025Quant-dLLM: Post-Training Extreme Low-Bit Quantization for Diffusion Large Language Modelscited by 11
  6. 2025PTQTP: Post-Training Quantization to Trit-Planes for Large Language Modelscited by 9
  7. 2024CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMscited by 7
  8. 2025ERC-SVD: Error-Controlled SVD for Large Language Model Compressioncited by 6
  9. 2025Layer-wise dynamic rank for compressing large language modelscited by 6
  10. 2025Progressive Binarization with Semi-Structured Pruning for LLMscited by 5
  11. 2025Event-Priori-Based Vision-Language Model for Efficient Visual Understandingcited by 5
  12. 2026CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMscited by 5
18 more
  1. 2025BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebookcited by 4
  2. 2025DBellQuant: Breaking the Bell with Double-Bell Transformation for LLMs Post Training Binarizationcited by 4
  3. 2025FlashEdit: Decoupling Speed, Structure, and Semantics for Precise Image Editingcited by 4
  4. 2025HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMscited by 4
  5. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 4
  6. 2025Highly Efficient and Effective LLMs with Multi-Boolean Architecturescited by 2
  7. 2025PT2-LLM: Post-Training Ternarization for Large Language Modelscited by 2
  8. 2026EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillationcited by 2
  9. 2025Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Modelscited by 1
  10. 2026D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMscited by 1
  11. 2026BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Modelscited by 1
  12. 2026NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Modelscited by 1
  13. 2026Fitting Is Not Enough: Smoothness in Extremely Quantized LLMscited by 1
  14. 2026SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Modelscited by 1
  15. 2025Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Modelscited by 0
  16. 2026RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMscited by 0
  17. 2026LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignmentcited by 0
  18. 2026One Qubit Can Beat One Bit: Quantum Advantage for Post-Training Quantizationcited by 0