mlsanityData to 14 Oct 2026

Addition is almost all you need: Compressing large language models with double binary factorization

arXiv 2505.11076 ↗First posted 16 May 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Binary quantization approaches, which replace weight matrices with binary matrices and substitute costly multiplications with cheaper additions, offer a computationally efficient approach to address the increasing computational and storage requirements of Large Language Models (LLMs). However, the severe quantization constraint (±1) can lead to significant accuracy degradation. In this paper, we propose Double Binary Factorization (DBF), a novel method that factorizes dense weight matrices into products of two binary (sign) matrices, each accompanied by scaling vectors. DBF preserves the efficiency advantages of binary representations while achieving compression rates that are competitive with or superior to state-of-the-art methods. Specifically, in a 1-bit per weight range, DBF is better than existing binarization approaches. In a 2-bit per weight range, DBF is competitive with the best quantization methods like QuIP# and QTIP. Unlike most existing compression techniques, which offer limited compression level choices, DBF allows fine-grained control over compression ratios by adjusting the factorization's intermediate dimension. Based on this advantage, we further introduce an algorithm for estimating non-uniform layer-wise compression ratios for DBF, based on previously developed channel pruning criteria. The code is available at: https://github.com/usamec/doublebinary

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

29 of 37 references.

References · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  12. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
17 more
  1. 2019Importance Estimation for Neural Network Pruningcited by 541
  2. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  3. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  4. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  5. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  6. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  7. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  8. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  9. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  10. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  11. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 70
  12. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 67
  13. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  14. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 59
  15. 2024Fast and Effective Weight Update for Pruned Large Language Modelscited by 11
  16. 2023Quantization Aware Factorization for Deep Neural Network Compressioncited by 6
  17. 2024Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorizationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026OneComp: One-Line Revolution for Generative AI Model Compressioncited by 3
  2. 2026Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuningcited by 0
  3. 2026From Sweep to Seam: Interleaved Cross-Block Post-Training Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization
  2. 2023PB-LLM: Partially Binarized Large Language Models
  3. 2024DB-LLM: Accurate Dual-Binarization for Efficient LLMs
  4. 2024BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
  5. 2024ARB-LLM: Alternating Refined Binarizations for Large Language Models
  6. 2026NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models
  7. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorization
  8. 2025When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models