More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization
Not on the map. Why?
Abstract (arXiv)
For extreme low-bit quantization of large language models (LLMs), Double Binary Factorization (DBF) is attractive as it enables efficient inference without sacrificing accuracy. However, the scaling parameters of DBF are too restrictive; after factoring out signs, all rank components share the same magnitude profile, resulting in performance saturation. We propose Multi-envelope DBF (MDBF), which retains a shared pair of 1-bit sign bases but replaces the single envelope with a rank-l envelope. By sharing sign matrices among envelope components, MDBF effectively maintains a binary carrier and utilizes the limited memory budget for magnitude expressiveness. We also introduce a closed-form initialization and an alternating refinement method to optimize MDBF. Across the LLaMA and Qwen families, MDBF enhances perplexity and zero-shot accuracy over previous binary formats at matched bits per weight while preserving the same deployment-friendly inference primitive.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 16 of its 22 references that are in the map. First posted on arXiv 31 Dec 2025.
Builds on
16 of 22 references.
References · 16
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
- 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
- 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
- 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
- 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
- 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
- 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
- 2023Controlling Continuous Relaxation for Combinatorial Optimizationcited by 16
- 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
4 more
- 2020Learning Quantized Neural Nets by Coarse Gradient Method for Non-linear Classificationcited by 8
- 2024Optimization by Parallel Quasi-Quantum Annealing with Gradient-Based Samplingcited by 6
- 2025LPCD: Unified Framework from Layer-Wise to Submodule Quantizationcited by 6
- 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimatorcited by 4
Built on by
Citing papers per year: 2026: 3.
3 papers.